DEV Community: balajivedagiri

Installing Openshift Cluster on vSphere7

balajivedagiri — Wed, 04 Oct 2023 05:27:59 +0000

Pre-requisites
Generate Pull secret from Redhat
Creating openshift cluster
Fixing Internal Image registry
Deploy a sample nginx application

1. Pre-requisites

a) Connectivity to vCenter on port 443 from openshift network.

b) Connectivity to ESXi hosts on port 443 from openshift network.

c) Generate ssh keys (we can use the existing), this needs to be passed during cluster creation.

d) Working DHCP for openshift cluster nodes.

e) Two static ip's for API and Apps, for Step d.

f) DNS entry for "api.." and "*.apps.." .

In our case we mapped as below in our DNS,
api.openshift-test01.tanzu.local => 192.168.144.22
*.apps.openshift-test01.tanzu.local => 192.168.144.23

2. Generate Pull secret from Redhat

Lets get the Pull secret and also download the installer and client tools.

a) Register with https://console.redhat.com/openshift/ using your personal email or official email.

b) Once logged in, Click on Create Cluster.

c) Choose "Datacenter" and scroll down

d) Click on vSphere

e) Click on Automated installation

f) Download the Installer, Pull secret, and Command line tools.

3. Creating openshift cluster

We use a linux jumpserver which is in same network as openshift network to create the cluster so the installer can connect to API server to verify the installation without any dependencies on Firewall.

root@linux-vm-automation:~/openshift# ls -ltr
total 414864
-rw-r--r-- 1 root root      2783 May 30 17:30 pull-secret.txt
-rw-r--r-- 1 root root  59819571 May 30 17:31 openshift-client-linux.tar.gz
-rw-r--r-- 1 root root 364993703 May 30 17:31 openshift-install-linux.tar.gz
root@linux-vm-automation:~/openshift#
root@linux-vm-automation:~/openshift#
root@linux-vm-automation:~/openshift# tar -xvf openshift-install-linux.tar.gz
README.md
openshift-install
root@linux-vm-automation:~/openshift# ll
total 975252
drwxr-xr-x  2 root root       146 May 30 18:04 ./
drwx------ 22 root root      4096 May 30 18:02 ../
-rw-r--r--  1 root root  59819571 May 30 17:31 openshift-client-linux.tar.gz
-rwxr-xr-x  1 root root 573825024 May  9 18:10 openshift-install*
-rw-r--r--  1 root root 364993703 May 30 17:31 openshift-install-linux.tar.gz
-rw-r--r--  1 root root      2783 May 30 17:30 pull-secret.txt
-rw-r--r--  1 root root       706 May  9 18:10 README.md
root@linux-vm-automation:~/openshift#

Parameters we passed to the installer are below, so ensure you have the details ready.

a) ssh public key.
b) select vsphere as platform.
c) vcenter ip address.
d) vcenter username and password with required previleges.
e) datacenter.
f) datastore.
g) network.
h) VIP for API and Ingress.
i) Domain Name.
j) cluster name.
k) enter the pull secret that we copied from redhat console.

root@linux-vm-automation:~/openshift# ./openshift-install create cluster
? SSH Public Key /root/.ssh/id_rsa.pub
? Platform vsphere
? vCenter 172.17.22.118
? Username administrator@vsphere.local
? Password [? for help] *************
INFO Connecting to vCenter 172.17.22.118
INFO Defaulting to only available datacenter: vcenter-datacenter
? Cluster tenant-cluster
? Default Datastore SSD_Storage
? Network tenant43-ntw-72a59d1a-398e-4018-8dbd-5afa8ca60d40
? Virtual IP Address for API 192.168.144.22
? Virtual IP Address for Ingress 192.168.144.23
? Base Domain tanzu.local
? Cluster Name openshift-test01
? Pull Secret [? for help] ******************************************************************************************************************************************************************************************************************INFO Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.13-9.2/builds/413.92.202305021736-0/x86_64/rhcos-413.92.202305021736-0-vmware.x86_64.ova?sha256='
INFO The file was found in cache: /root/.cache/openshift-installer/image_cache/rhcos-413.92.202305021736-0-vmware.x86_64.ova. Reusing...
INFO Creating infrastructure resources...
INFO Waiting up to 20m0s (until 8:22AM) for the Kubernetes API at https://api.openshift-test01.tanzu.local:6443...
INFO API v1.26.3+b404935 up
INFO Waiting up to 30m0s (until 8:35AM) for bootstrapping to complete...
INFO Destroying the bootstrap resources...
INFO Waiting up to 40m0s (until 9:05AM) for the cluster at https://api.openshift-test01.tanzu.local:6443 to initialize...
INFO Checking to see if there is a route at openshift-console/console...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/openshift/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.openshift-test01.tanzu.local
INFO Login to the console with user: "kubeadmin", and password: "c9T8a-ALwe9-ZU7D2-ENTDh"
INFO Time elapsed: 44m32s
root@linux-vm-automation:~/openshift#

So Cluster is created, lets login and verify.

Installer above provided the url and credentials to login

INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/openshift/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.openshift-test01.tanzu.local
INFO Login to the console with user: "kubeadmin", and password: "c9T8a-ALwe9-ZU7D2-ENTDh"
INFO Time elapsed: 44m32s

Access the cluster using oc or kubectl,

We already download oc tool "openshift-client-linux.tar.gz" from the redhat console, extract it and place it in /usr/local/bin/ or the location that you prefer.

export KUBECONFIG=/root/openshift/auth/kubeconfig

root@linux-vm-automation:~/openshift# oc get nodes
NAME                                    STATUS   ROLES                  AGE   VERSION
openshift-test01-pg8s9-master-0         Ready    control-plane,master   35m   v1.26.3+b404935
openshift-test01-pg8s9-master-1         Ready    control-plane,master   35m   v1.26.3+b404935
openshift-test01-pg8s9-master-2         Ready    control-plane,master   34m   v1.26.3+b404935
openshift-test01-pg8s9-worker-0-5c42f   Ready    worker                 14m   v1.26.3+b404935
openshift-test01-pg8s9-worker-0-djzl5   Ready    worker                 15m   v1.26.3+b404935
openshift-test01-pg8s9-worker-0-mtgzh   Ready    worker                 14m   v1.26.3+b404935
root@linux-vm-automation:~/openshift#

4. Fixing Internal Image registry

In vSphere environment, Openshift Internal Image registry won't be available since shareable stroage ReadWriteMany can't be created on vSphere storage.

If you try to create a pod with image pointing to internal image registry,

It will fail like below,

To Fix it, first create a PVC

root@linux-vm-automation:~/openshift# cat openshift-image-registry-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: image-registry-storage
  namespace: openshift-image-registry
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
root@linux-vm-automation:~/openshift#

root@linux-vm-automation:~/openshift# oc create -f openshift-image-registry-pvc.yaml -n openshift-image-registry
persistentvolumeclaim/image-registry-storage created
root@linux-vm-automation:~/openshift#

Update the Registry CR spec with the following command.

oc edit configs.imageregistry.operator.openshift.io -n openshift-image-registry

Change spec.managementState from Removed to Managed.
Change spec.storage from {} to: claim: image-registry-storage

spec:
    managementState: Managed
storage:
      pvc:
        claim: image-registry-storage

After updating it should look like below,

spec:
    managementState: Managed
storage:
      pvc:
        claim: image-registry-storage

Once image registry pod is running fine, images from the internal image registry should be available.

The example which was not running earlier is running now,

5. Deploy a sample nginx application.

You should already be familiar on how to deploy a pod. Below we created a deployment using nginx image and created a service.

Lets create a route in openshift,

Note : This is not a Kubernetes object like Service.

Provisioning an RKE2 (Rancher Kubernetes Engine 2) cluster on vSphere

balajivedagiri — Sun, 04 Jun 2023 23:07:22 +0000

In this article i will walk you down with steps to create RKE2 cluster on vSphere vCenter from Rancher UI.

Pre-requisites

Rancher nodes needs to communicate with vSphere vCenter on port 443.
Rancher nodes needs to communicate with RKE2 cluster nodes on port 22.

1. Installing packages in the template VM.

Create a new Ubuntu VM, perform below steps and later convert it into a template. We will let rancher use this template to create VM's.

Ensure below packages are installed in the template,
• curl
• wget
• git
• net-tools
• unzip
• apparmor-parser
• ca-certificates
• cloud-init
• cloud-guest-utils
• cloud-image-utils
• growpart
• cloud-initramfs-growroot
• open-iscsi
• openssh-server
• open-vm-tools



sudo apt-get update
sudo apt-get install -y curl wget git net-tools unzip ca-certificates cloud-init cloud-guest-utils cloud-image-utils cloud-initramfs-growroot open-iscsi openssh-server open-vm-tools net-tools apparmor

2. Configure the datasource for cloud-init in the template VM.

Rancher will use cloud-init for things like setting hostname, creating a user, running a script, etc.
Set the datasource for cloud-init using command “dpkg-reconfigure cloud-init”.



sudo dpkg-reconfigure cloud-init

And ensure “NoCloud” datasource is selected like below, I have deselected all other datasources since my requirement for rancher is only “NoCloud”.

Verify that changes are propagated to the config file,



root@:~# cat /etc/cloud/cloud.cfg.d/90_dpkg.cfg
# to update this file, run dpkg-reconfigure cloud-init
datasource_list: [ NoCloud ]
root@:~#

3. Convert the VM into a template.

Run below script to scrub the VM ( similar to sysprep on Windows ).
Save below contents to a file and execute it.



#!/bin/bash
# Cleaning logs.
if [ -f /var/log/audit/audit.log ]; then
  cat /dev/null > /var/log/audit/audit.log
fi
if [ -f /var/log/wtmp ]; then
  cat /dev/null > /var/log/wtmp
fi
if [ -f /var/log/lastlog ]; then
  cat /dev/null > /var/log/lastlog
fi

# Cleaning udev rules.
if [ -f /etc/udev/rules.d/70-persistent-net.rules ]; then
  rm /etc/udev/rules.d/70-persistent-net.rules
fi

# Cleaning the /tmp directories
rm -rf /tmp/*
rm -rf /var/tmp/*

# Cleaning the SSH host keys
rm -f /etc/ssh/ssh_host_*

# Cleaning the machine-id
truncate -s 0 /etc/machine-id
rm /var/lib/dbus/machine-id
ln -s /etc/machine-id /var/lib/dbus/machine-id

# Cleaning the shell history
unset HISTFILE
history -cw
echo > ~/.bash_history
rm -fr /root/.bash_history

# Truncating hostname, hosts, resolv.conf and setting hostname to localhost
truncate -s 0 /etc/{hostname,hosts,resolv.conf}
hostnamectl set-hostname localhost

# Clean cloud-init
cloud-init clean -s -l

Now clone the VM to a template.

4. Add vSphere vCenter credentials in Rancher.

Click Create and Click on VMware vSphere

Enter your vSphere vCenter credentials, we will use administrator account. For granular permissions, please refer rancher documentation.

5. Create RKE2 Cluster in vSphere

Go back to Rancher homepage, Click on Create.

Ensure you toggle the switch to RKE2 as highlighted below and click on VMware vSphere.

Enter the details,

Pool1, we will use to create "Control Plane Nodes". Ensure you select appropriate Data Center/Resource Pool/Data Store/Folder.

Select the Template that you created in the initial steps, along with CPU/Memory/Networks/etc like below.

Add another pool for worker node,

Fill the information like we did above for control plane nodes.

For the sake simplicity, we will keep the default values for the cluster and Click Create,

Cluster creation in progress,

Wait for the nodes to be bootstrapped and cluster creation,

If you see VM settings, rancher would have mounted an iso called user-data.iso

If you login to one of the node and navigate to /mnt, it will have user-data and meta-data and used by cloud-init (if you remember, this is the reason, we selected NoCloud as data source for cloud-init)



root@vsphere-rke2-test01-pool1-4a86ea2d-tf5jq:~# cd /mnt

root@vsphere-rke2-test01-pool1-4a86ea2d-tf5jq:/mnt# ls

meta-data  user-data

root@vsphere-rke2-test01-pool1-4a86ea2d-tf5jq:/mnt# cat meta-data

hostname: vsphere-rke2-test01-pool1-4a86ea2d-tf5jq

root@vsphere-rke2-test01-pool1-4a86ea2d-tf5jq:/mnt#

root@vsphere-rke2-test01-pool1-4a86ea2d-tf5jq:/mnt# cat user-data

  
  
  cloud-config


groups:


staff
hostname: vsphere-rke2-test01-pool1-4a86ea2d-tf5jq
runcmd:
sh /usr/local/custom_script/install.sh
set_hostname:
vsphere-rke2-test01-pool1-4a86ea2d-tf5jq
users:
create_groups: false
groups: staff
lock_passwd: true
name: docker
no_user_group: true

Add a new worker node.

Lets add a new worker node to existing cluster.
Go to Rancher home => Click burger menu => Click on Cluster Management => Click on your cluster.

Under pool2 dedicated to worker nodes, click on Plus icon,

events on vcenter of node getting created,

Worker node2 successfully added.

Installing and Configuring Elasticsearch/Kibana 8.x with Security

balajivedagiri — Wed, 10 May 2023 13:17:02 +0000

We will be installing,configuring elasticsearch and kibana 8.4, but steps should be same for most versions.

Our cluster will have 3 master nodes, 3 hot data nodes, 3 warm data nodes and 1 machine learning node.

1) pre-requisites

1a) create /var/lib/elasticsearch mount point on all the nodes.
1b) turn off swap on OS(to ensure JVM heap is not swapped out).
1c) since we are using packages to install elasticsearch, ulimits are enforced in systemd unit file /usr/lib/systemd/system/elasticsearch.service.
1d) settings like file descriptors, max processes, max virtual memory size , max file size, etc are controlled from the systemd unit file.
1e) change default value of TCP retransmission timeout value, update the net.ipv4.tcp_retries2 setting in /etc/sysctl.conf to 5, and sysctl -w net.ipv4.tcp_retries2=5.

2) Installing elasticsearch

Our cluster will have 3 master nodes, 3 hot data nodes, 3 warm data nodes and 1 machine learning node.

2a) Import elasticsearch PGP key.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

2b) Install apt-transport-https package

sudo apt install apt-transport-https

2c) save the repo,

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

2d) update the repo and install the package,

apt update && apt install elasticsearch

apt-get install elasticsearch
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 119 not upgraded.
Need to get 0 B/566 MB of archives.
After this operation, 1,170 MB of additional disk space will be used.
Selecting previously unselected package elasticsearch.
(Reading database ... 111616 files and directories currently installed.)
Preparing to unpack .../elasticsearch_8.4.3_amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (8.4.3) ...
Setting up elasticsearch (8.4.3) ...
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : B25meUI2L6WcfTWBNvNp

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

2e) Ansible playbook to install the package.

---
- hosts: elasticsearch
  become: true
  gather_facts: true
  tasks:
  - name: Import the Elasticsearch PGP key
    apt_key:
      url: https://artifacts.elastic.co/GPG-KEY-elasticsearch
      keyring: /usr/share/keyrings/elasticsearch-keyring.gpg
      state: present
  - name: Install apt-transport-https
    apt:
      name: apt-transport-https
      state: present
# Add elasticsearch repo into sources list file /etc/apt/sources.list.d/elastic-8.x.list, after adding it will also run apt update or apt-get update by default
  - apt_repository:
      repo: 'deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main'
      state: present
      filename: elastic-8.x.list
  - name: Install a specific version of elasticsearch
    apt:
      name: elasticsearch=8.4.3
      state: present
      update_cache: yes

2f) enable the service to start automatically on boot

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch

3) Generating certificates to enable TLS for transport and http.

3a) Generate CA certificate.

Login to one of the node where you installed elasticsearch , and issue below command to generate CA certificate. For higher protection, ensure you are setting password the certificate when it prompts below at the end and ensure you save that password in a secure location to use it later.

/usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /root/elasticsearch_certs/elasticsearch-test-ca.p12

/usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /root/elasticsearch_certs/elasticsearch-test-ca.p12
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Enter password for elasticsearch-test-ca.p12 :

root@jumperserver:~/elasticsearch_certs# ls
elasticsearch-test-ca.p12

3b) Generate node certificates

We use node certificates to join nodes to cluster and for transport layer encrytion. add all of your node details with dns name and ip into an yaml file like below,

root@jumperserver:~# cat /root/elasticsearch_certs/instances.yaml
instances:
  - name: "test-elastic-master01"
    ip: "10.10.4.6"
    dns: "test-elastic-master01"
  - name: "test-elastic-master02"
    ip: "10.10.4.7"
    dns: "test-elastic-master02"
  - name: "test-elastic-master03"
    ip: "10.10.4.8"
    dns: "test-elastic-master03"
  - name: "test-elastic-hotdata01"
    ip: "10.10.4.2"
    dns: "test-elastic-hotdata01"
  - name: "test-elastic-hotdata02"
    ip: "10.10.4.3"
    dns: "test-elastic-hotdata02"
  - name: "test-elastic-hotdata03"
    ip: "10.10.4.4"
    dns: "test-elastic-hotdata03"
  - name: "test-elastic-warmdata01"
    ip: "10.10.4.11"
    dns: "test-elastic-warmdata01"
  - name: "test-elastic-warmdata02"
    ip: "10.10.4.12"
    dns: "test-elastic-warmdata02"
  - name: "test-elastic-warmdata03"
    ip: "10.10.4.13"
    dns: "test-elastic-warmdata03"
  - name: "test-elastic-ml01"
    ip: "10.10.4.10"
    dns: "test-elastic-ml01"

below you need to enter CA certificate password that you entered in step 3a, and ensure you set password for each and every node certificate ( you can set same password for all the nodes or different password as per security compliance)

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --in /root/elasticsearch_certs/instances.yaml --out /root/elasticsearch_certs/server-cert-bundle.zip --ca /root/elasticsearch_certs/elasticsearch-test-ca.p12

root@elasticsearch-jumperserver:~/elasticsearch_certs# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --in /root/elasticsearch_certs/instances.yaml --out /root/elasticsearch_certs/server-cert-bundle.zip --ca /root/elasticsearch_certs/elasticsearch-test-ca.p12
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X.509 certificate and private keys.
    * By default, this generates a single certificate and key for use
       on a single instance.
    * The '-multiple' option will prompt you to enter details for multiple
       instances and will generate a certificate and key for each one
    * The '-in' option allows for the certificate generation to be automated by describing
       the details of each instance in a YAML file

    * An instance is any piece of the Elastic Stack that requires an SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.


    * All certificates generated by this tool will be signed by a certificate authority (CA)
      unless the --self-signed command line option is specified.
      The tool can automatically generate a new CA for you, or you can provide your own with
      the --ca or --ca-cert command line options.


By default the 'cert' mode produces a single PKCS#12 output file which holds:
    * The instance certificate
    * The private key for the instance certificate
    * The CA certificate

If you specify any of the following options:
    * -pem (PEM formatted output)
    * -multiple (generate multiple certificates)
    * -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files

Enter password for CA (/root/elasticsearch_certs/elasticsearch-test-ca.p12) :
Enter password for test-elastic-master01/test-elastic-master01.p12 :
Enter password for test-elastic-master02/test-elastic-master02.p12 :
Enter password for test-elastic-master03/test-elastic-master03.p12 :
Enter password for test-elastic-hotdata01/test-elastic-hotdata01.p12 :
Enter password for test-elastic-hotdata02/test-elastic-hotdata02.p12 :
Enter password for test-elastic-hotdata03/test-elastic-hotdata03.p12 :
Enter password for test-elastic-warmdata01/test-elastic-warmdata01.p12 :
Enter password for test-elastic-warmdata02/test-elastic-warmdata02.p12 :
Enter password for test-elastic-warmdata03/test-elastic-warmdata03.p12 :
Enter password for test-elastic-ml01/test-elastic-ml01.p12 :

Certificates written to /root/elasticsearch_certs/server-cert-bundle.zip

This file should be properly secured as it contains the private keys for
all instances
After unzipping the file, there will be a directory for each instance.
Each instance has a single PKCS#12 (.p12) file containing the instance
certificate, instance private key and the CA certificate
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
root@elasticsearch-jumperserver:~/elasticsearch_certs#

below we are checking the generated certificates,

root@elasticsearch-jumperserver:~/elasticsearch_certs# ls
elasticsearch-test-ca.p12  instances.yaml  server-cert-bundle.zip
root@elasticsearch-jumperserver:~/elasticsearch_certs#

root@elasticsearch-jumperserver:~/elasticsearch_certs# unzip server-cert-bundle.zip
Archive:  server-cert-bundle.zip
   creating: test-elastic-master01/
  inflating: test-elastic-master01/test-elastic-master01.p12
   creating: test-elastic-master02/
  inflating: test-elastic-master02/test-elastic-master02.p12
   creating: test-elastic-master03/
  inflating: test-elastic-master03/test-elastic-master03.p12
   creating: test-elastic-hotdata01/
  inflating: test-elastic-hotdata01/test-elastic-hotdata01.p12
   creating: test-elastic-hotdata02/
  inflating: test-elastic-hotdata02/test-elastic-hotdata02.p12
   creating: test-elastic-hotdata03/
  inflating: test-elastic-hotdata03/test-elastic-hotdata03.p12
   creating: test-elastic-warmdata01/
  inflating: test-elastic-warmdata01/test-elastic-warmdata01.p12
   creating: test-elastic-warmdata02/
  inflating: test-elastic-warmdata02/test-elastic-warmdata02.p12
   creating: test-elastic-warmdata03/
  inflating: test-elastic-warmdata03/test-elastic-warmdata03.p12
   creating: test-elastic-ml01/
  inflating: test-elastic-ml01/test-elastic-ml01.p12
root@elasticsearch-jumperserver:~/elasticsearch_certs#
root@elasticsearch-jumperserver:~/elasticsearch_certs# ls
elasticsearch-test-ca.p12  test-elastic-hotdata02  test-elastic-master01  test-elastic-master03  test-elastic-warmdata01  test-elastic-warmdata03  server-cert-bundle.zip
test-elastic-hotdata01      test-elastic-hotdata03  test-elastic-master02  test-elastic-ml01      test-elastic-warmdata02  instances.yaml
root@elasticsearch-jumperserver:~/elasticsearch_certs#
root@elasticsearch-jumperserver:~/elasticsearch_certs# ls -ltr *
-rw-r--r-- 1 root root   876 Oct 26 18:49 instances.yaml
-rw------- 1 root root  2672 Oct 26 18:55 elasticsearch-test-ca.p12
-rw------- 1 root root 39406 Oct 26 18:56 server-cert-bundle.zip

test-elastic-master01:
total 4
-rw-r--r-- 1 root root 3700 Oct 26 18:56 test-elastic-master01.p12

test-elastic-master02:
total 4
-rw-r--r-- 1 root root 3700 Oct 26 18:56 test-elastic-master02.p12

test-elastic-master03:
total 4
-rw-r--r-- 1 root root 3700 Oct 26 18:56 test-elastic-master03.p12

test-elastic-hotdata01:
total 4
-rw-r--r-- 1 root root 3702 Oct 26 18:56 test-elastic-hotdata01.p12

test-elastic-hotdata03:
total 4
-rw-r--r-- 1 root root 3702 Oct 26 18:56 test-elastic-hotdata03.p12

test-elastic-hotdata02:
total 4
-rw-r--r-- 1 root root 3702 Oct 26 18:56 test-elastic-hotdata02.p12

test-elastic-warmdata01:
total 4
-rw-r--r-- 1 root root 3704 Oct 26 18:56 test-elastic-warmdata01.p12

test-elastic-warmdata03:
total 4
-rw-r--r-- 1 root root 3704 Oct 26 18:56 test-elastic-warmdata03.p12

test-elastic-warmdata02:
total 4
-rw-r--r-- 1 root root 3704 Oct 26 18:56 test-elastic-warmdata02.p12

test-elastic-ml01:
total 4
-rw-r--r-- 1 root root 3676 Oct 26 18:56 test-elastic-ml01.p12
root@elasticsearch-jumperserver:~/elasticsearch_certs#

3c) Generate http certificate.

generate http certificates for http encryption, ensure you enter hostnames and ip's of the machines from which you would like you to communicate with elaticsearch over http, e.g jumpservers, kibana, elasticsearch nodes , so on.

/usr/share/elasticsearch/bin/elasticsearch-certutil http

root@elasticsearch-jumperserver:~/elasticsearch_certs/http# /usr/share/elasticsearch/bin/elasticsearch-certutil http

## Elasticsearch HTTP Certificate Utility

The 'http' command guides you through the process of generating certificates
for use on the HTTP (Rest) interface for Elasticsearch.

This tool will ask you a number of questions in order to generate the right
set of files for your needs.

## Do you wish to generate a Certificate Signing Request (CSR)?

A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you don't have
access to the keys for that CA).

If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.

If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.

Generate a CSR? [y/N]n

## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?

If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.

If you do not have an existing CA, one will be generated for you.

Use an existing CA? [y/N]y

## What is the path to your CA?

Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
CA Path: /root/elasticsearch_certs/elasticsearch-test-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore's password to be blank,
in which case you can simply press <ENTER> at the prompt
Password for elasticsearch-test-ca.p12:

## How long should your certificates be valid?

Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.

Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.

You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)

For how long should your certificate be valid? [5y] 10y

## Do you wish to generate one certificate per node?

If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.

Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.

If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.

However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.

Generate a certificate per node? [y/N]N

## Which hostnames will be used to connect to your nodes?

These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.

You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.

If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.

Enter all the hostnames that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

test-elastic-master01
test-elastic-master02
test-elastic-master03
test-elastic-kibana01
test-elastic-clustmon01
elasticsearch-jumpserver

You entered the following hostnames.

 - test-elastic-master01
 - test-elastic-master02
 - test-elastic-master03
 - test-elastic-kibana01
 - test-elastic-clustmon01
 - elasticsearch-jumpserver

Is this correct [Y/n]Y

## Which IP addresses will be used to connect to your nodes?

If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.

If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press <ENTER> to skip this step.

Enter all the IP addresses that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

10.10.4.6
10.10.4.7
10.10.4.8
10.10.4.5
10.10.4.16
10.10.4.17
10.10.4.18
10.10.4.1
10.10.4.31

You entered the following IP addresses.

 - 10.10.4.6
 - 10.10.4.7
 - 10.10.4.8
 - 10.10.4.5
 - 10.10.4.16
 - 10.10.4.17
 - 10.10.4.18
 - 10.10.4.1
 - 10.10.4.31

Is this correct [Y/n]Y

## Other certificate options

The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.

Key Name: test-elastic-master01
Subject DN: CN=test-elastic-master01
Key Size: 2048

Do you wish to change any of these options? [y/N]N

## What password do you want for your private key(s)?

Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.

If you wish to use a blank password, simply press <enter> at the prompt below.
Provide a password for the "http.p12" file:  [<ENTER> for none]
Repeat password to confirm:

## Where should we save the generated files?

A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.

These files will be included in a single zip archive.

What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]

Zip file written to /usr/share/elasticsearch/elasticsearch-ssl-http.zip
root@elasticsearch-jumperserver:~/elasticsearch_certs/http#

root@elasticsearch-jumperserver:~/elasticsearch_certs/http# ls
elasticsearch  elasticsearch-ssl-http.zip  kibana
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# mv elasticsearch-ssl-http.zip elasticsearch-ssl-http.zip_old
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# mv elasticsearch elasticsearch_old
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# mv kibana kibana_old
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# pwd
/root/elasticsearch_certs/http
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# cp /usr/share/elasticsearch/elasticsearch-ssl-http.zip .
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# unzip elasticsearch-ssl-http.zip
Archive:  elasticsearch-ssl-http.zip
   creating: elasticsearch/
  inflating: elasticsearch/README.txt
  inflating: elasticsearch/http.p12
  inflating: elasticsearch/sample-elasticsearch.yml
   creating: kibana/
  inflating: kibana/README.txt
  inflating: kibana/elasticsearch-ca.pem
  inflating: kibana/sample-kibana.yml
root@elasticsearch-jumperserver:~/elasticsearch_certs/http#
root@elasticsearch-jumperserver:~/elasticsearch_certs/http# ls
elasticsearch  elasticsearch_old  elasticsearch-ssl-http.zip  elasticsearch-ssl-http.zip_old  kibana  kibana_old
root@elasticsearch-jumperserver:~/elasticsearch_certs/http#

4) Copy the generated certificates

Copy the node certificate and http certificate to respective nodes to the path /etc/elasticsearch/certs/

Note: Node certificate is different for each and every elasticsearch node, http certificate is common for all the nodes.

5) Setting keystore and trustore for transport and http

Transport Truststore password is the password of CA certificate.
Transport Keystore password is the password of node certificates.

Transport http password is the password of http certificate.

set transport truststore/keystore and http keystore with below commands on all the nodes, you need run below commands on each and every elasticsearch nodes,

/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

6) Configuring elasticsearch parameters

Setting the configuration in /etc/elasticsearch/elasticsearch.yml, comment all the existing lines and append below after changing ip and hostnames to your node ip's and hostnames,

6a) Master nodes

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
cluster.name: test-elasticsearch
node.name: test-elastic-master01
network.host: 10.10.4.6
discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
node.roles: [ master ]
xpack.watcher.enabled: true


# transport SSL/TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/test-elastic-master01.p12
xpack.security.transport.ssl.truststore.path: certs/test-elastic-master01.p12

# http SSL/TLS
http.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12

6b) Hot nodes

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
cluster.name: test-elasticsearch
node.name: test-elastic-hotdata01
network.host: 10.10.4.2
discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
node.roles: [ data,ingest ]
node.attr.box_type: hot
xpack.watcher.enabled: true


# transport SSL/TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/test-elastic-hotdata01.p12
xpack.security.transport.ssl.truststore.path: certs/test-elastic-hotdata01.p12

# http SSL/TLS
http.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12

6c) Warm nodes

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
cluster.name: test-elasticsearch
node.name: test-elastic-warmdata01
network.host: 10.10.4.11
discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
node.roles: [ data,ingest ]
node.attr.box_type: warm
xpack.watcher.enabled: true


# transport SSL/TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/test-elastic-warmdata01.p12
xpack.security.transport.ssl.truststore.path: certs/test-elastic-warmdata01.p12
# http SSL/TLS
http.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12

6d) ML nodes

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
cluster.name: test-elasticsearch
node.name: test-elastic-ml01
network.host: 10.10.4.10
discovery.seed_hosts: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]
node.roles: [ ml ]
xpack.watcher.enabled: true


# transport SSL/TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/test-elastic-ml01.p12
xpack.security.transport.ssl.truststore.path: certs/test-elastic-ml01.p12
# http SSL/TLS
http.host: 0.0.0.0
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12

7) Starting elasticsearch

Start the nodes one by one using systemctl start elasticsearch, you can monitor the logs in /var/log/elasticsearch/test-elasticsearch.log

We need to ensure we remove the paramter once cluster is formed in /etc/elasticsearch/elasticsearch.yml

cluster.initial_master_nodes: ["test-elastic-master01", "test-elastic-master02", "test-elastic-master03"]

8) resetting elastic user password

you can also do this once you start the first node,

root@test-elastic-master01:/var/log/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
This tool will reset the password of the [elastic] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y


Password for the [elastic] user successfully reset.
New value: xxxxxxxxxxxxxxxxxxxxxxxx
root@test-elastic-master01:/var/log/elasticsearch#

9) Check the status of cluster and list nodes

root@test-elastic-master01:/var/log/elasticsearch# curl -X GET "https://10.10.4.2:9200/_cluster/health?pretty"  -u elastic -k
Enter host password for user 'elastic':
{
  "cluster_name" : "test-elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 6,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 2,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}


root@test-elastic-master01:/var/log/elasticsearch# curl -X GET "https://10.10.4.2:9200/_cat/nodes?pretty"  -u elastic -k
Enter host password for user 'elastic':
10.10.4.2  2 63 0 0.04 0.05 0.02 di - test-elastic-hotdata01
10.10.4.3  2 63 0 0.00 0.06 0.06 di - test-elastic-hotdata02
10.10.4.8  7 97 1 0.00 0.10 0.09 m  - test-elastic-master03
10.10.4.7 11 96 2 0.00 0.03 0.01 m  * test-elastic-master02
10.10.4.6 10 97 2 0.00 0.04 0.02 m  - test-elastic-master01
10.10.4.4  2 62 0 0.00 0.06 0.05 di - test-elastic-hotdata03
root@test-elastic-master01:/var/log/elasticsearch#

10) Install and Configure Kibana

10a) Installing kibana

root@test-elastic-kibana01:~# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

root@test-elastic-kibana01:~# apt-get install apt-transport-https
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  apt-transport-https
1 upgraded, 0 newly installed, 0 to remove and 118 not upgraded.
Need to get 1,704 B of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B]
Fetched 1,704 B in 1s (3,407 B/s)
(Reading database ... 111616 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_2.0.9_all.deb ...
Unpacking apt-transport-https (2.0.9) over (2.0.8) ...
Setting up apt-transport-https (2.0.9) ...
root@test-elastic-kibana01:~#

root@test-elastic-kibana01:~# echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main
root@test-elastic-kibana01:~#

root@test-elastic-kibana01:~# sudo apt-get update && sudo apt-get install kibana
0% [Working]
Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease
Get:2 https://artifacts.elastic.co/packages/8.x/apt stable InRelease [10.4 kB]
Hit:3 http://us.archive.ubuntu.com/ubuntu focal InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Get:5 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 Packages [34.0 kB]
Hit:6 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
Get:7 https://artifacts.elastic.co/packages/8.x/apt stable/main i386 Packages [3,556 B]
Fetched 48.0 kB in 1s (33.1 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  kibana
0 upgraded, 1 newly installed, 0 to remove and 118 not upgraded.
Need to get 285 MB of archives.
After this operation, 680 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 kibana amd64 8.4.3 [285 MB]
Fetched 285 MB in 3s (83.2 MB/s)
Selecting previously unselected package kibana.
(Reading database ... 111616 files and directories currently installed.)
Preparing to unpack .../kibana_8.4.3_amd64.deb ...
Unpacking kibana (8.4.3) ...
Setting up kibana (8.4.3) ...
Creating kibana group... OK
Creating kibana user... OK
Created Kibana keystore in /etc/kibana/kibana.keystore
root@test-elastic-kibana01:~#

10b) Copy the ca certificate to kibana server

Copy the ca certificate that was generated from the step 3c kibana/elasticsearch-ca.pem to /etc/kibana/elasticsearch-ca.pem

10c) Reset kibana_system password

To do below, login into one of the elasticsearch node which is added to http certificate.

root@test-elastic-master01:/var/log/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system
This tool will reset the password of the [kibana_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y

Password for the [kibana_system] user successfully reset.
New value: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
root@test-elastic-master01:/var/log/elasticsearch#

10d) Configuring kibana

set below parameters in /etc/kibana/kibana.yml, we are pointing to hot data nodes below

elasticsearch.hosts: ["https://10.10.4.2:9200","https://10.10.4.3:9200","https://10.10.4.4:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "xxxxxxxxxxxxxxxxxxxxxxxx"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elasticsearch-ca.pem" ]

10f) Start kibana and enable the service

systemctl start kibana
systemctl enable kibana

Access kibana using elastic user using url http://kibana-hostname:5601

Using cloud-init to customize a VM while provisioning with terraform on vSphere

balajivedagiri — Thu, 02 Mar 2023 03:45:59 +0000

We are going to perform below steps on a newly installed ubuntu VM and convert it to a template

1) Check if VMware tools is installed,



root@linux:~# dpkg -l | grep open-vm-tools
ii  open-vm-tools                          2:11.3.0-2ubuntu0~ubuntu20.04.4   amd64        Open VMware Tools for virtual machines hosted on VMware (CLI)
root@linux:~#

2) Update VMware tools,



root@linux:~# apt install open-vm-tools
Reading package lists... Done
Building dependency tree
Reading state information... Done
open-vm-tools is already the newest version (2:11.3.0-2ubuntu0~ubuntu20.04.4).
The following packages were automatically installed and are no longer required:
  linux-headers-5.15.0-46-generic linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic linux-hwe-5.15-headers-5.15.0-46 linux-image-5.15.0-46-generic linux-image-5.4.0-137-generic
  linux-modules-5.15.0-46-generic linux-modules-5.4.0-137-generic linux-modules-extra-5.15.0-46-generic linux-modules-extra-5.4.0-137-generic
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@linux:~#

3) Check the version installed,



root@linux:~# vmtoolsd -v
VMware Tools daemon, version 11.3.0.29534 (build-18090558)
root@linux:~#

4) Ensure the open-vm-tools service is enabled and started,



root@linux:~# systemctl enable open-vm-tools.service
Synchronizing state of open-vm-tools.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable open-vm-tools
root@linux:~#
root@linux:~# systemctl start open-vm-tools.service
root@linux:~#
root@linux:~# systemctl is-enabled open-vm-tools.service
enabled
root@linux:~#
root@linux:~# systemctl status open-vm-tools.service
● open-vm-tools.service - Service for virtual machines hosted on VMware
     Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-03-02 06:08:57 +04; 9min ago
       Docs: http://open-vm-tools.sourceforge.net/about.php
   Main PID: 710 (vmtoolsd)
      Tasks: 3 (limit: 9350)
     Memory: 4.3M
     CGroup: /system.slice/open-vm-tools.service
             └─710 /usr/bin/vmtoolsd

Mar 02 06:08:57 linux systemd[1]: Started Service for virtual machines hosted on VMware.
root@linux:~#

5) Install cloud-init package,



root@linux:~# dpkg -l | grep -w cloud-init
root@linux:~#
root@linux:~# apt install cloud-init
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-5.15.0-46-generic linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic linux-hwe-5.15-headers-5.15.0-46 linux-image-5.15.0-46-generic linux-image-5.4.0-137-generic
  linux-modules-5.15.0-46-generic linux-modules-5.4.0-137-generic linux-modules-extra-5.15.0-46-generic linux-modules-extra-5.4.0-137-generic
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  eatmydata libeatmydata1 python3-distutils python3-importlib-metadata python3-json-pointer python3-jsonpatch python3-jsonschema python3-lib2to3 python3-more-itertools python3-pyrsistent
  python3-setuptools python3-zipp
Suggested packages:
  python-jsonschema-doc python-setuptools-doc
The following NEW packages will be installed:
  cloud-init eatmydata libeatmydata1 python3-distutils python3-importlib-metadata python3-json-pointer python3-jsonpatch python3-jsonschema python3-lib2to3 python3-more-itertools
  python3-pyrsistent python3-setuptools python3-zipp
0 upgraded, 13 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,260 kB of archives.
After this operation, 7,347 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

root@linux:~# dpkg -l | grep -w cloud-init
ii  cloud-init                             22.4.2-0ubuntu0~20.04.2           all          initialization and customization tool for cloud instances
root@linux:~#

6) enable the cloud-init service,
it doesn't need to be started, but ensure it is enabled,



root@linux:~# systemctl status cloud-init.service
● cloud-init.service - Initial cloud-init job (metadata service crawler)
     Loaded: loaded (/lib/systemd/system/cloud-init.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
root@linux:~#
root@linux:~# systemctl enable cloud-init.service
root@linux:~#
root@linux:~# sudo systemctl is-enabled cloud-init.service
enabled
root@linux:~#

7) Ensure no datasources are enabled in /etc/cloud/cloud.cfg like below,



root@linux:~# cat /etc/cloud/cloud.cfg | grep datasource
# If you use datasource_list array, keep array items in a single line.
# Example datasource config
# datasource:
root@linux:~#

8) Check the current activated datasources,



root@linux:/etc/cloud/cloud.cfg.d# cat /etc/cloud/cloud.cfg.d/90_dpkg.cfg
# to update this file, run dpkg-reconfigure cloud-init
datasource_list: [ NoCloud, ConfigDrive, OpenNebula, DigitalOcean, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, SmartOS, Bigstep, Scaleway, AliYun, Ec2, CloudStack, Hetzner, IBMCloud, Oracle, Exoscale, RbxCloud, UpCloud, VMware, Vultr, LXD, NWCS, None ]
root@linux:/etc/cloud/cloud.cfg.d#

9) Run the command "dpkg-reconfigure cloud-init" and select only VMware (this is not mandatory , but deselecting other datasources will reduce our boot time, so cloud-init doesn't need to look for the userdata/metadata from all the datasources in the list)



root@linux:/etc/cloud/cloud.cfg.d# dpkg-reconfigure cloud-init
root@linux:/etc/cloud/cloud.cfg.d#
root@linux:/etc/cloud/cloud.cfg.d# cat /etc/cloud/cloud.cfg.d/90_dpkg.cfg
# to update this file, run dpkg-reconfigure cloud-init
datasource_list: [ VMware ]
root@linux:/etc/cloud/cloud.cfg.d#

10) ensure /etc/cloud/cloud.cfg.d has only below files,



root@linux:/etc/cloud/cloud.cfg.d# ls
05_logging.cfg  90_dpkg.cfg  README
root@linux:/etc/cloud/cloud.cfg.d#

11) disable network, so cloud-init doesn't configure networking,



root@linux:/etc/cloud/cloud.cfg.d# vi /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
root@linux:/etc/cloud/cloud.cfg.d#
root@linux:/etc/cloud/cloud.cfg.d# cat /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
network: {config: disabled}
root@linux:/etc/cloud/cloud.cfg.d#

12) As a final step, run the clean command to ensure that cloud-init will all the modules in the userdata and metadata.



root@linux:/# cloud-init clean --logs
root@linux:/#

13) below we are verifying that niginx/apache2 are not installed on the vm before converting it to a template,



root@linux:~# dpkg -l | grep nginx
root@linux:~#
root@linux:~# dpkg -l | grep apache2
root@linux:~#

14) Convert the VM to a template from vCenter

15) Provision a VM using terraform code which can be obtained from below repo,

https://github.com/balajivedagiri/terraform_cloud-init.git

userdata we defined it,

cloud-config

runcmd:

[ sh, -c, echo "=========Hello There from Terraform and Cloud-init automation=========" > /root/testing-01] packages:
- nginx
- apache2

16) Once VM is provisioned, login to the new VM and verify if cloud-init ran successfully.



root@cloud-init-testing02:~# cloud-init status
status: running
root@cloud-init-testing02:~#

# Above it is still running, wait for sometime and check
# Below it completed successfully

root@cloud-init-testing02:~# cloud-init status
status: done
root@cloud-init-testing02:~#

17) Check if packages are installed successfully,



root@cloud-init-testing02:~# dpkg -l | grep nginx
ii  libnginx-mod-http-image-filter         1.18.0-0ubuntu1.4                 amd64        HTTP image filter module for Nginx
ii  libnginx-mod-http-xslt-filter          1.18.0-0ubuntu1.4                 amd64        XSLT Transformation module for Nginx
ii  libnginx-mod-mail                      1.18.0-0ubuntu1.4                 amd64        Mail module for Nginx
ii  libnginx-mod-stream                    1.18.0-0ubuntu1.4                 amd64        Stream module for Nginx
ii  nginx                                  1.18.0-0ubuntu1.4                 all          small, powerful, scalable web/proxy server
ii  nginx-common                           1.18.0-0ubuntu1.4                 all          small, powerful, scalable web/proxy server - common files
ii  nginx-core                             1.18.0-0ubuntu1.4                 amd64        nginx web/proxy server (standard version)
root@cloud-init-testing02:~#
root@cloud-init-testing02:~# dpkg -l | grep apache2
ii  apache2                                2.4.41-4ubuntu3.13                amd64        Apache HTTP Server
ii  apache2-bin                            2.4.41-4ubuntu3.13                amd64        Apache HTTP Server (modules and other binary files)
ii  apache2-data                           2.4.41-4ubuntu3.13                all          Apache HTTP Server (common files)
ii  apache2-utils                          2.4.41-4ubuntu3.13                amd64        Apache HTTP Server (utility programs for web servers)
root@cloud-init-testing02:~#
root@cloud-init-testing02:~# cat /root/testing-01
=========Hello There from Terraform and Cloud-init automation=========
root@cloud-init-testing02:~#
root@cloud-init-testing02:~# ls -ltr /root/testing-01
-rw-r--r-- 1 root root 71 Mar  2 07:15 /root/testing-01
root@cloud-init-testing02:~#

18) to check what userdata we have passed,



root@cloud-init-testing02:~# cloud-init query userdata
#cloud-config
runcmd:
  - [ sh, -c, echo "=========Hello There from Terraform and Cloud-init automation=========" > /root/testing-01]
packages:
- nginx
- apache2
root@cloud-init-testing02:~#
root@cloud-init-testing02:~# vmware-rpctool "info-get guestinfo.userdata" | base64 -d
#cloud-config
runcmd:
  - [ sh, -c, echo "=========Hello There from Terraform and Cloud-init automation=========" > /root/testing-01]
packages:
- nginx
- apache2

root@cloud-init-testing02:~#

Installing Rancher on a highly available RKE2 cluster

balajivedagiri — Sun, 11 Sep 2022 19:13:43 +0000

Rancher is a complete software stack for teams adopting containers. It addresses the operational and security challenges of managing multiple Kubernetes clusters across any infrastructure.

Below is a sample rancher architecture.

We are going to install a 3 node rke2 kubernetes cluster and install rancher in the rke2 cluster using Helm.

1. Create a private Loadbalancer

Create a L4 loadbalancer with the first rke2 node as backendpool( once rest of the two nodes are added to the rke2 cluster, you can add the 2 rke2 nodes to loadbalancer backendpool), and below are ports that loadbalancer and backend traffic should be listening on,

1) 9345 to register new nodes
2) 6443 for Kubernetes API Server

2. Maintain the host entries on the rke2 nodes like below

maintain a hostname/dns for the loadbalancer vip that is created in the first setup. Here i am pointing dns/hostname "rke2.mydomain.ae" to my loadbalancer vip 182.17.12.5

172.17.11.11 rancher01
172.17.11.12 rancher02
172.17.11.13 rancher03
172.17.12.5 rke2.mydomain.ae

3. Launch the first server node

3a. adding hostnames or IP for tls

To avoid certificate errors with the fixed registration address, you should launch the server with the tls-san parameter set. This option adds an additional hostname or IP as a Subject Alternative Name in the server's TLS cert, and it can be specified as a list if you would like to access via both the IP and the hostname.



root@rancher01:~# mkdir -p /etc/rancher/rke2/

Note: below i maintained all my nodes ip's, hostnames and including loadbalancer dns name that points to loadbalancer vip that was created in the first step.



root@rancher01:~# cat /etc/rancher/rke2/config.yaml
tls-san:
  - rke2.mydomain.ae
  - 172.17.11.11
  - rancher01
  - 172.17.11.12
  - rancher02
  - 172.17.11.13
  - rancher03
root@rancher01:~#

3b. Installing rke2

Here i am pulling the latest stable rke2 release.



root@rancher01:~# curl -sfL https://get.rke2.io | sh -
[INFO]  finding release for channel stable
[INFO]  using v1.23.9+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.23.9+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.23.9+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@rancher01:~#



root@rancher01:~# systemctl start rke2-server.service

Note: When the rke2-server service is started for the first time , it will take few minutes as it needs to pull all the images need to spin up the cluster. So don't panic.
If you want, you can monitor the setup process using the command "journalctl -u rke2-server -f"



root@rancher01:~# systemctl status rke2-server.service
● rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/usr/local/lib/systemd/system/rke2-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-08-25 18:22:05 +04; 1min 15s ago
       Docs: https://github.com/rancher/rke2#readme
    Process: 235133 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 235135 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 235136 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 235138 (rke2)
      Tasks: 181
     Memory: 3.4G
     CGroup: /system.slice/rke2-server.service
             ├─235138 /usr/local/bin/rke2 server
             ├─235177 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/a>
             ├─235290 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=f>

Note:

After running this installation rke2-server service will be installed. The rke2-server service will be configured to automatically restart after node reboots or if the process crashes or is killed.

Additional utilities will be installed at /var/lib/rancher/rke2/bin/. They include: kubectl, crictl, and ctr. Note that these are not on your path by default.

Two cleanup scripts will be installed to the path at /usr/local/bin/rke2. They are: rke2-killall.sh and rke2-uninstall.sh.

A kubeconfig file will be written to /etc/rancher/rke2/rke2.yaml.

A token that can be used to register other server or agent nodes will be created at /var/lib/rancher/rke2/server/node-token

3c. Accessing the cluster

rke2 will automatically download the kubectl needed, i will be available in below location.



root@rancher01:~# ll /var/lib/rancher/rke2/bin/
total 315140
drwxr-xr-x 2 root root       176 Aug 25 18:20 ./
drwxr-xr-x 4 root root        31 Aug 25 18:20 ../
-rwxr-xr-x 1 root root  54096264 Aug 25 18:20 containerd*
-rwxr-xr-x 1 root root   7369488 Aug 25 18:20 containerd-shim*
-rwxr-xr-x 1 root root  11527464 Aug 25 18:20 containerd-shim-runc-v1*
-rwxr-xr-x 1 root root  11539944 Aug 25 18:20 containerd-shim-runc-v2*
-rwxr-xr-x 1 root root  35018008 Aug 25 18:20 crictl*
-rwxr-xr-x 1 root root  20463560 Aug 25 18:20 ctr*
-rwxr-xr-x 1 root root  49328448 Aug 25 18:20 kubectl*
-rwxr-xr-x 1 root root 122372296 Aug 25 18:20 kubelet*
-rwxr-xr-x 1 root root  10961304 Aug 25 18:20 runc*



root@rancher01:~# cp /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/

root@rancher01:~# chmod +x /usr/local/bin/kubectl



root@rancher01:~# cat /etc/rancher/rke2/rke2.yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    client-key-data: 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Set the KUBECONFIG env variable as specified below



root@rancher01:~# export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
root@rancher01:~#
root@rancher01:~# kubectl get nodes -o wide
NAME               STATUS   ROLES                       AGE     VERSION          INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
rancher01   Ready    control-plane,etcd,master   3m31s   v1.23.9+rke2r1   172.17.11.11   <none>        Ubuntu 20.04.4 LTS   5.4.0-100-generic   containerd://1.5.13-k3s1
root@rancher01:~#



root@rancher01:~# kubectl get all -A
NAMESPACE     NAME                                                        READY   STATUS      RESTARTS   AGE
kube-system   pod/cloud-controller-manager-rancher01               1/1     Running     0          5m24s
kube-system   pod/etcd-rancher01                                   1/1     Running     0          5m21s
kube-system   pod/helm-install-rke2-canal-kdm5c                           0/1     Completed   0          5m12s
kube-system   pod/helm-install-rke2-coredns-jnzgh                         0/1     Completed   0          5m12s
kube-system   pod/helm-install-rke2-ingress-nginx-bsxkp                   0/1     Completed   0          5m12s
kube-system   pod/helm-install-rke2-metrics-server-8vn8f                  0/1     Completed   0          5m12s
kube-system   pod/kube-apiserver-rancher01                         1/1     Running     0          4m59s
kube-system   pod/kube-controller-manager-rancher01                1/1     Running     0          5m25s
kube-system   pod/kube-proxy-rancher01                             1/1     Running     0          5m23s
kube-system   pod/kube-scheduler-rancher01                         1/1     Running     0          4m48s
kube-system   pod/rke2-canal-vkr74                                        2/2     Running     0          4m57s
kube-system   pod/rke2-coredns-rke2-coredns-545d64676-s7hhs               1/1     Running     0          4m57s
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-5dd676f5c7-fvdhw   1/1     Running     0          4m57s
kube-system   pod/rke2-ingress-nginx-controller-67zjf                     1/1     Running     0          4m11s
kube-system   pod/rke2-metrics-server-6564db4569-vllzm                    1/1     Running     0          4m29s

NAMESPACE     NAME                                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes                                ClusterIP   10.43.0.1       <none>        443/TCP         5m28s
kube-system   service/rke2-coredns-rke2-coredns                 ClusterIP   10.43.0.10      <none>        53/UDP,53/TCP   4m58s
kube-system   service/rke2-ingress-nginx-controller-admission   ClusterIP   10.43.188.253   <none>        443/TCP         4m11s
kube-system   service/rke2-metrics-server                       ClusterIP   10.43.43.187    <none>        443/TCP         4m29s

NAMESPACE     NAME                                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/rke2-canal                      1         1         1       1            1           kubernetes.io/os=linux   4m57s
kube-system   daemonset.apps/rke2-ingress-nginx-controller   1         1         1       1            1           kubernetes.io/os=linux   4m11s

NAMESPACE     NAME                                                   READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/rke2-coredns-rke2-coredns              1/1     1            1           4m58s
kube-system   deployment.apps/rke2-coredns-rke2-coredns-autoscaler   1/1     1            1           4m58s
kube-system   deployment.apps/rke2-metrics-server                    1/1     1            1           4m29s

NAMESPACE     NAME                                                              DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/rke2-coredns-rke2-coredns-545d64676               1         1         1       4m58s
kube-system   replicaset.apps/rke2-coredns-rke2-coredns-autoscaler-5dd676f5c7   1         1         1       4m58s
kube-system   replicaset.apps/rke2-metrics-server-6564db4569                    1         1         1       4m29s

NAMESPACE     NAME                                         COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-rke2-canal            1/1           17s        5m23s
kube-system   job.batch/helm-install-rke2-coredns          1/1           17s        5m23s
kube-system   job.batch/helm-install-rke2-ingress-nginx    1/1           68s        5m23s
kube-system   job.batch/helm-install-rke2-metrics-server   1/1           46s        5m23s
root@rancher01:~#

4. Adding Second server node to the cluster

4a. Setting up the Second server node

First copy the node token generated from the first server node as highlighted below.



root@rancher01:/var/lib/rancher# cat /var/lib/rancher/rke2/server/node-token
K10d11c154bab23851058711225726a1189ba7f00642b87c82e0b7407cdfc25c82d::server:2ffddc9f0d8901c2b6e30bde043850e1
root@rancher01:/var/lib/rancher#



root@rancher01:~# mkdir -p /etc/rancher/rke2/

Maintain the config.yaml as below,



root@rancher01:~# cat /etc/rancher/rke2/config.yaml
token: K10d11c154bab23851058711225726a1189ba7fasfsafalsfasf8a8f9saf::server:2ffddc9f0dasdfasf9a898980
server: https://rke2.mydomain.ae:9345
tls-san:
  - rke2.mydomain.ae
  - 172.17.11.11
  - rancher01
  - 172.17.11.12
  - rancher02
  - 172.17.11.13
  - rancher03

Ensure host entry on the second server node is setup similar to first server node.



cat /etc/hosts
172.17.11.11    rancher01
172.17.11.12    rancher02
172.17.11.13    rancher03
172.16.132.35   rke2.mydomain.ae

4b. Installing rke2 on the second server node



root@rancher02:~# curl -sfL https://get.rke2.io | sh -
[INFO]  finding release for channel stable
[INFO]  using v1.23.9+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.23.9+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.23.9+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@rancher02:~#

we ran into an issue. Second server nodes didn't start, below is the error



journalctl -u rke2-server -f

Aug 25 18:56:40 rancher02 systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Aug 25 18:56:45 rancher02 systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 15.
Aug 25 18:56:45 rancher02 systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Aug 25 18:56:45 rancher02 systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Aug 25 18:56:45 rancher02 sh[195790]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Aug 25 18:56:45 rancher02 sh[195798]: /bin/sh: 1: /usr/bin/systemctl: not found
Aug 25 18:56:45 rancher02 rke2[195804]: time="2022-08-25T18:56:45+04:00" level=warning msg="not running in CIS mode"
Aug 25 18:56:45 rancher02 rke2[195804]: time="2022-08-25T18:56:45+04:00" level=info msg="Starting rke2 v1.23.9+rke2r1 (2d206eba8d0180351408dbed544c852b6b4fdd42)"
Aug 25 18:57:05 rancher02 rke2[195804]: time="2022-08-25T18:57:05+04:00" level=fatal msg="starting kubernetes: preparing server: failed to get CA certs: Get \"https://rke2.mydomain.ae:9345/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 25 18:57:05 rancher02 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Aug 25 18:57:05 rancher02 systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Aug 25 18:57:05 rancher02 systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).

As per the log above, it says failed to get CA certs: Get \"https://rke2.mydomain.ae:9345/cacerts\"

When we accessed the same cacerts using first node hostname, it works. It returns the certificate.



root@rancher01:~# curl https://rancher01:9345/cacerts -k
-----BEGIN CERTIFICATE-----
cnZlci1jYUAxNjYxNDM4NDI1MB4XDTIyMDgyNTE0NDAyNVoXDTMyMDgyMjE0NDA
dc5sEBfQUANmtPK4ckGrMYpYmFT5EAyBMnmoNRB0brnfxFCjQjBAMA4GA1UdDwEB
NVowJDEiMCAGA1UEAwwZcmtlMi1zZXJ2ZXItY2FAMTY2MTQzODQyNTBZMBMGByqG
SM49AgEGCCqGSM49AwEHA0IABG7NRoHKS8bDW1IZZE2gGxGrEYCDUfvWtSk/xw3R
dc5sEBfQUANmtPK4ckGrMYpYmFT5EAyBMnmoNRB0brnfxFCjQjBAMA4GA1UdDwEB
/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQ/5eGLG0uix3SOpwhc
pDJr95aaEzAKBggqhkjOPQQDAgNIADBFAiEA26zz5tif+FH7UT6VbJp8ig631yMV
APBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQ/5eGLG0uix3SOpwhc
-----END CERTIFICATE-----
root@rancher01:~#



root@rancher01:~# curl https://rke2.mydomain.ae:9345/cacerts
^C

we are able to get the CA cert using node ip or hostname but not with vip. There was an issue in loadbalancer config, after correcting the config in loadbalancer.



root@rancher01:~# curl https://rke2.mydomain.ae:9345/cacerts -k
-----BEGIN CERTIFICATE-----
cnZlci1jYUAxNjYxNDM4NDI1MB4XDTIyMDgyNTE0NDAyNVoXDTMyMDgyMjE0NDA
dc5sEBfQUANmtPK4ckGrMYpYmFT5EAyBMnmoNRB0brnfxFCjQjBAMA4GA1UdDwEB
NVowJDEiMCAGA1UEAwwZcmtlMi1zZXJ2ZXItY2FAMTY2MTQzODQyNTBZMBMGByqG
SM49AgEGCCqGSM49AwEHA0IABG7NRoHKS8bDW1IZZE2gGxGrEYCDUfvWtSk/xw3R
dc5sEBfQUANmtPK4ckGrMYpYmFT5EAyBMnmoNRB0brnfxFCjQjBAMA4GA1UdDwEB
/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQ/5eGLG0uix3SOpwhc
pDJr95aaEzAKBggqhkjOPQQDAgNIADBFAiEA26zz5tif+FH7UT6VbJp8ig631yMV
APBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQ/5eGLG0uix3SOpwhc
-----END CERTIFICATE-----
root@rancher01:~#



root@rancher02:~# systemctl status rke2-server.service
● rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/usr/local/lib/systemd/system/rke2-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-08-25 19:10:15 +04; 1min 13s ago
       Docs: https://github.com/rancher/rke2#readme
    Process: 198178 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 198180 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 198181 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 198182 (rke2)
      Tasks: 142
     Memory: 3.7G
     CGroup: /system.slice/rke2-server.service
             ├─198182 /usr/local/bin/rke2 server
             ├─198192 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/a>

root@rancher02:~#

4c.Access the cluster from first server node



root@rancher01:~# kubectl get nodes -o wide
NAME               STATUS   ROLES                       AGE    VERSION          INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
rancher01   Ready    control-plane,etcd,master   30m    v1.23.9+rke2r1   172.17.11.11   <none>        Ubuntu 20.04.4 LTS   5.4.0-100-generic   containerd://1.5.13-k3s1
rancher02   Ready    control-plane,etcd,master   2m8s   v1.23.9+rke2r1   172.17.11.12   <none>        Ubuntu 20.04.4 LTS   5.4.0-100-generic   containerd://1.5.13-k3s1
root@rancher01:~#

5. Adding third server node to the cluster.



root@rancher03:~# mkdir -p /etc/rancher/rke2/



root@rancher03:~# cat /etc/rancher/rke2/config.yaml

token: K10d11c154bab23851058711225726a1189ba7fasfsafalsfasf8a8f9saf::server:2ffddc9f0dasdfasf9a898980
server: https://rke2.mydomain.ae:9345
tls-san:
  - rke2.mydomain.ae
  - 172.17.11.11
  - rancher01
  - 172.17.11.12
  - rancher02
  - 172.17.11.13
  - rancher03



cat /etc/hosts
172.17.11.11    rancher01
172.17.11.12    rancher02
172.17.11.13    rancher03
172.16.132.35   rke2.mydomain.ae



root@rancher03:~# curl -sfL https://get.rke2.io | sh -
[INFO]  finding release for channel stable
[INFO]  using v1.23.9+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.23.9+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.23.9+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@rancher03:~#



root@rancher03:~# systemctl status rke2-server.service
● rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/usr/local/lib/systemd/system/rke2-server.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: https://github.com/rancher/rke2#readme
root@rancher03:~#



root@rancher03:~# systemctl enable rke2-server.service
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service.
root@rancher03:~#



root@rancher03:~# systemctl start rke2-server.service
root@rancher03:~#



root@rancher01:~# kubectl get nodes -o wide
NAME               STATUS   ROLES                       AGE   VERSION          INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
rancher01   Ready    control-plane,etcd,master   39m   v1.23.9+rke2r1   172.17.11.11   <none>        Ubuntu 20.04.4 LTS   5.4.0-100-generic   containerd://1.5.13-k3s1
rancher02   Ready    control-plane,etcd,master   11m   v1.23.9+rke2r1   172.17.11.12   <none>        Ubuntu 20.04.4 LTS   5.4.0-100-generic   containerd://1.5.13-k3s1
rancher03   Ready    control-plane,etcd,master   54s   v1.23.9+rke2r1   172.17.11.13   <none>        Ubuntu 20.04.4 LTS   5.4.0-100-generic   containerd://1.5.13-k3s1
root@rancher01:~#

6. Installing Rancher using HELM on the rke2 3 node cluster.



root@rancher01:~# wget https://get.helm.sh/helm-v3.9.4-linux-amd64.tar.gz
--2022-08-25 19:37:08--  https://get.helm.sh/helm-v3.9.4-linux-amd64.tar.gz
Resolving get.helm.sh (get.helm.sh)... 152.199.21.175, 2606:2800:233:1cb7:261b:1f9c:2074:3c
Connecting to get.helm.sh (get.helm.sh)|152.199.21.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14026634 (13M) [application/x-tar]
Saving to: ‘helm-v3.9.4-linux-amd64.tar.gz’

helm-v3.9.4-linux-amd64.tar.gz                  100%[====================================================================================================>]  13.38M  --.-KB/s    in 0.06s

2022-08-25 19:37:08 (233 MB/s) - ‘helm-v3.9.4-linux-amd64.tar.gz’ saved [14026634/14026634]



root@rancher01:~# tar -zxvf helm-v3.8.2-linux-amd64.tar.gz
linux-amd64/
linux-amd64/helm
linux-amd64/LICENSE
linux-amd64/README.md



root@rancher01:~# cd linux-amd64/

root@rancher01:~/linux-amd64# ls
helm  LICENSE  README.md



root@rancher01:~/linux-amd64# cp -pr helm /usr/local/bin/
root@rancher01:~/linux-amd64# which helm
/usr/local/bin/helm



root@rancher01:~# helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

"rancher-stable" has been added to your repositories
root@rancher01:~#



root@rancher01:~# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "rancher-stable" chart repository
Update Complete. ⎈Happy Helming!⎈
root@rancher01:~#



root@rancher01:~# kubectl create namespace cattle-system
namespace/cattle-system created
root@rancher01:~#

Below we have set tls=external since we will be installing the tls/ssl certificates in the loadbalancer. You can setup according to your needs. Refer rancher documentation below for more information.

https://docs.ranchermanager.rancher.io/pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster



root@rancher01:~# helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=rancher.mydomain.com --set bootstrapPassword='mypassword123' --set tls=external
NAME: rancher
LAST DEPLOYED: Thu Aug 25 19:57:37 2022
NAMESPACE: cattle-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Rancher Server has been installed.

NOTE: Rancher may take several minutes to fully initialize. Please standby while Certificates are being issued, Containers are started and the Ingress rule comes up.

Check out our docs at https://rancher.com/docs/

If you provided your own bootstrap password during installation, browse to https://rancher.mydomain.com to get started.

If this is the first time you installed Rancher, get started by running this command and clicking the URL it generates:


echo https://rancher.mydomain.com/dashboard/?setup=$(kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}')


To get just the bootstrap password on its own, run:


kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{ "\n" }}'



Happy Containering!
root@rancher01:~#



root@rancher01:~# kubectl get all -n cattle-system
NAME                           READY   STATUS              RESTARTS   AGE
pod/rancher-75b7b67cbb-lrh7z   0/1     ContainerCreating   0          9s
pod/rancher-75b7b67cbb-nqt85   0/1     ContainerCreating   0          9s
pod/rancher-75b7b67cbb-rjggf   0/1     ContainerCreating   0          9s

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/rancher   ClusterIP   10.43.251.67   <none>        80/TCP,443/TCP   9s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rancher   0/3     3            0           9s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/rancher-75b7b67cbb   3         3         0       9s
root@rancher01:~#

After few mins,



root@rancher01:~# kubectl get all -n cattle-system
NAME                           READY   STATUS    RESTARTS   AGE
pod/rancher-75b7b67cbb-lrh7z   1/1     Running   0          93s
pod/rancher-75b7b67cbb-nqt85   1/1     Running   0          93s
pod/rancher-75b7b67cbb-rjggf   1/1     Running   0          93s

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/rancher   ClusterIP   10.43.251.67   <none>        80/TCP,443/TCP   93s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rancher   3/3     3            3           93s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/rancher-75b7b67cbb   3         3         3       93s
root@rancher01:~#

To login to the UI, use the dns name that was using during the helm installation. For testing you can point the dns/hostname to the loadbalancer ip or any node ip.

https://rancher.mydomain.com or http://rancher.mydomain.com

DEV Community: balajivedagiri

Installing Openshift Cluster on vSphere7

Contents

1. Pre-requisites

2. Generate Pull secret from Redhat

3. Creating openshift cluster

4. Fixing Internal Image registry

5. Deploy a sample nginx application.

Provisioning an RKE2 (Rancher Kubernetes Engine 2) cluster on vSphere

Pre-requisites

1. Installing packages in the template VM.

2. Configure the datasource for cloud-init in the template VM.

3. Convert the VM into a template.

4. Add vSphere vCenter credentials in Rancher.

5. Create RKE2 Cluster in vSphere

cloud-config

Add a new worker node.

Installing and Configuring Elasticsearch/Kibana 8.x with Security

1) pre-requisites

2) Installing elasticsearch

3) Generating certificates to enable TLS for transport and http.

4) Copy the generated certificates

*5) Setting keystore and trustore for transport and http *

6) Configuring elasticsearch parameters

7) Starting elasticsearch

8) resetting elastic user password

9) Check the status of cluster and list nodes

10) Install and Configure Kibana

Using cloud-init to customize a VM while provisioning with terraform on vSphere

cloud-config

Installing Rancher on a highly available RKE2 cluster

1. Create a private Loadbalancer

2. Maintain the host entries on the rke2 nodes like below

3. Launch the first server node

4. Adding Second server node to the cluster

5. Adding third server node to the cluster.

6. Installing Rancher using HELM on the rke2 3 node cluster.

5) Setting keystore and trustore for transport and http