OpenShift Installation Using UPI on VMware vSphere Machines

Anzal Beg — Mon, 15 Jul 2024 11:35:13 +0000

Table of Contents

Introduction
Prerequisites
Required Hardware Specifications
User-Provisioned DNS Requirements
Load Balancing Requirements for User-Provisioned Infrastructure
Validating DNS Resolution
Generating a Key Pair for SSH Access
Obtaining the Installation Program
Download the RHCOS images
Installing the OpenShift CLI
Creating the Installation Configuration File
Creating the Kubernetes Manifest and Ignition Config Files
Creating a Web Server on the same Installation VM/Different VM
Copy the ignition files to the Web Server
Boot ISO Image Preparation Using Custom ISO and Ansible Automation
Login to the Cluster
Conclusion**

Introduction

Red Hat OpenShift is a powerful Kubernetes platform for automating the deployment, scaling, and management of containerized applications. This guide provides a step-by-step process to install OpenShift v4.14 on a vSphere environment. By following these instructions, you can set up a robust and scalable infrastructure for your containerized applications.

Prerequisites

Before you begin the installation, ensure you have the following:

Necessary hardware specifications met.
Access to the vSphere environment to upload the ISO file.
Red Hat OpenShift Cluster Manager account to access the download key.
Necessary DNS and load balancing configurations.
SSH access to the cluster nodes.
HAProxy installed and configured on the load balancer node.

Required Hardware Specifications

For an OpenShift installation on VMware using User-Provisioned Infrastructure (UPI), the typical node/vm requirements are as follows:

Bootstrap Node:

Count: 1
Purpose: Used for the initial bootstrap process of the OpenShift cluster. It is removed after the installation is complete.
Requirements:
CPU: 4 vCPUs
Memory: 16 GB RAM
Storage: 120 GB

Control Plane (Master) Nodes:

Count: 3 (Recommended for high availability)
Purpose: Manage the OpenShift cluster and its components.
Requirements:
CPU: 4 vCPUs
Memory: 16 GB RAM
Storage: 120 GB

Compute (Worker) Nodes:

Count: At least 2 (Can be scaled based on workload requirements)
Purpose: Run the user workloads (applications).
Requirements:
CPU: 2-4 vCPUs (Depending on the workload)
Memory: 8-16 GB RAM (Depending on the workload)
Storage: 120 GB

Infrastructure Nodes (Optional):

Count: As needed (Typically 2-3 for large clusters)
Purpose: Dedicated to running infrastructure-related workloads like registry, monitoring, logging, etc.
Requirements:
CPU: 4 vCPUs
Memory: 16 GB RAM
Storage: 120 GB

Example for a Basic Setup:

Bootstrap Node: 1
Control Plane Nodes: 3
Compute Nodes: 2
Summary of VMs:
Total Nodes: 6
Total vCPUs: 28-32
Total Memory: 88-96 GB
Total Storage: 720 GB

Example of Production Setup:

To successfully install OpenShift, the following hardware specifications are required:

Machine	Memory	CPU	Storage	Notes	OS
Bootstrap	16 GB	4	50 GB	Used only during installation	RHCOS
Helper Machine	8 GB	4	50 GB	Web server for deployment purposes	Ubuntu
HA Proxy Node #1	16 GB	4	50 GB	Load Balancer node	Ubuntu
HA Proxy Node #2	16 GB	4	50 GB	Load Balancer node	Ubuntu
Controller Node #1	32 GB	16	120 GB		RHCOS
Controller Node #2	32 GB	16	120 GB		RHCOS
Controller Node #3	32 GB	16	120 GB		RHCOS
Worker Nodes #1	64 GB	16	120 GB		RHCOS
Worker Nodes #2	64 GB	16	120 GB		RHCOS
Worker Nodes #3	64 GB	16	120 GB		RHCOS
Worker Nodes #4	64 GB	16	120 GB		RHCOS
Worker Nodes #5	64 GB	16	120 GB		RHCOS
Infra Node #1	32 GB	10	120 GB + 2 TB (HDD/SSD)		RHCOS
Infra Node #2	32 GB	10	120 GB + 2 TB (HDD/SSD)		RHCOS
Infra Node #3	32 GB	10	120 GB + 2 TB (HDD/SSD)		RHCOS

User-Provisioned DNS Requirements

Ensure that your DNS and hostnames are configured as follows:

DNS/Hostname	IP	IP Description
api.openshift.onlinecluster.com	174.168.68.57	F5 VIP Pointing to the internal load balancer haproxy servers
api-int.openshift.onlinecluster.com	174.168.68.57	F5 VIP Pointing to the internal load balancer haproxy servers
*.apps.openshift.onlinecluster.com	174.168.68.57	F5 VIP Pointing to the internal load balancer haproxy servers
oocbs01.openshift.onlinecluster.com	174.168.95.70	Bootstrap VM IP
ooccn01.openshift.onlinecluster.com	174.168.95.74	Master Node VM IP
ooccn02.openshift.onlinecluster.com	174.168.95.75	Master Node VM IP
ooccn03.openshift.onlinecluster.com	174.168.95.74	Master Node VM IP
oocinfra01.openshift.onlinecluster.com	174.168.95.74	Worker Node Role Infra IP
oocinfra02.openshift.onlinecluster.com	174.168.95.74	Worker Node Role Infra IP
oocinfra03.openshift.onlinecluster.com	174.168.95.74	Worker Node Role Infra IP
oocwn01.openshift.onlinecluster.com	174.168.95.74	Worker Node IP
oocwn02.openshift.onlinecluster.com	174.168.95.74	Worker Node IP
oocwn03.openshift.onlinecluster.com	174.168.95.74	Worker Node IP
oocwn04.openshift.onlinecluster.com	174.168.95.74	Worker Node IP
oocwn05.openshift.onlinecluster.com	174.168.95.74	Worker Node IP

Load Balancing Requirements for User-Provisioned Infrastructure

Proper load balancing is essential for high availability and scalability. Here are the load balancer configurations:

API Load Balancer

Install HAProxy on your HAProxy node using the following commands:

sudo apt update && sudo apt upgrade
sudo apt install haproxy

Edit the HAProxy configuration file:

sudo nano /etc/haproxy/haproxy.cfg

Add the following configuration:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend api
    bind 174.168.95.72:6443
    default_backend api
    mode tcp

frontend machine-api
    bind 174.168.95.72:22623
    default_backend machine-api
    mode tcp

backend api
    balance roundrobin
    mode tcp
    server OOCCN01 ooccn01.openshift.onlinecluster.com:6443 check
    server OOCCN02 ooccn02.openshift.onlinecluster.com:6443 check
    server OOCCN03 ooccn03.openshift.onlinecluster.com:6443 check
    server OOCBS01 oocbs01.openshift.onlinecluster.com:6443 check

backend machine-api
    balance roundrobin
    mode tcp
    server OOCCN01 ooccn01.openshift.onlinecluster.com:22623 check
    server OOCCN02 ooccn02.openshift.onlinecluster.com:22623 check
    server OOCCN03 ooccn03.openshift.onlinecluster.com:22623 check
    server OOCBS01 oocbs01.openshift.onlinecluster.com:22623 check

Ingress Load Balancer

Edit the HAProxy configuration file:

sudo nano /etc/haproxy/haproxy.cfg

Add the following configuration:

frontend ingress-http
    bind 174.168.95.72:80
    default_backend ingress-http
    mode tcp

frontend ingress-https
    bind 174.168.95.72:443
    default_backend ingress-https
    mode tcp

backend ingress-http
    balance roundrobin
    mode tcp
    server OOCWN01 oocwn01.openshift.onlinecluster.com:80 check
    server OOCWN02 oocwn02.openshift.onlinecluster.com:80 check
    server OOCWN03 oocwn03.openshift.onlinecluster.com:80 check
    server OOCWN04 oocwn04.openshift.onlinecluster.com:80 check
    server OOCWN05 oocwn05.openshift.onlinecluster.com:80 check

backend ingress-https
    balance roundrobin
    mode tcp
    server OOCWN01 oocwn01.openshift.onlinecluster.com:443 check
    server OOCWN02 oocwn02.openshift.onlinecluster.com:443 check
    server OOCWN03 oocwn03.openshift.onlinecluster.com:443 check
    server OOCWN04 oocwn04.openshift.onlinecluster.com:443 check
    server OOCWN05 oocwn05.openshift.onlinecluster.com:443 check

Validating DNS Resolution

Ensure that DNS is properly resolving. Use the following commands to validate:

dig +noall +answer 174.168.1.1 api.openshift.onlinecluster.com 
api.openshift.onlinecluster.com.   3600    IN      A       174.168.68.57

dig +noall +answer 174.168.1.1 .apps.openshift.onlinecluster.com 
.apps.openshift.onlinecluster.com. 3600 IN    A       174.168.1.1

dig +noall +answer 174.168.1.1 oocbs01.ajsb.ajman.ae
oocbs01.openshift.onlinecluster.com. 3600 IN A       174.168.95.70
dig +noall +answer  174.168.1.1 -x 174.168.68.57
57.68.168.174.in-addr.arpa. 3600 IN     PTR     *.apps.openshift.onlinecluster.com.
57.68.168.174.in-addr.arpa. 3600 IN     PTR     api-int.openshift.onlinecluster.com.
57.68.168.174.in-addr.arpa. 3600 IN     PTR     api.openshift.onlinecluster.com.
57.68.168.174.in-addr.arpa. 3600 IN     PTR     *.apps.openshift.onlinecluster.com.

Generating a Key Pair for SSH Access

Generate an SSH key pair for access to the nodes:

ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/id_rsa

This command generates a public and private key pair. Ensure the public key (~/.ssh/id_rsa.pub) is added to the nodes.

Obtaining the Installation Program

To download the OpenShift installer:

export OCP_RELEASE=4.14.0
export RELEASE_IMAGE=quay.io/openshift-release-dev/ocp-release:$OCP_RELEASE-x86_64
curl -o openshift-install-linux.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$OCP_RELEASE/openshift-install-linux-$OCP_RELEASE.tar.gz
tar -zxvf openshift-install-linux.tar.gz -C /usr/local/bin

Installing the OpenShift CLI

To install the oc command-line tool:

curl -o openshift-client-linux.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$OCP_RELEASE/openshift-client-linux-$OCP_RELEASE.tar.gz
tar -zxvf openshift-client-linux.tar.gz -C /usr/local/bin

Download your installation pull secret from the Red Hat OpenShift Cluster Manager https://console.redhat.com/openshift/install/pull-secret.

Download the RHCOS Images

curl -o rhcos-4.14.0-x86_64-live.x86_64.iso  https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.14/4.14.0/rhcos-4.14.0-x86_64-live.x86_64.iso    
curl -o rhcos-4.14.0-x86_64-metal.x86_64.raw.gz https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.14/4.14.0/rhcos-4.14.0-x86_64-metal.x86_64.raw.gz

Creating the Installation Configuration File

Create a configuration file for the installation:
Create a new directory:

mkdir -p /home/user/ocp-install/
cd /home/user/ocp-install
mv /<download-dir>/ openshift-install .
openshift-install create install-config --dir=.

Edit the generated install-config.yaml to match your environment settings, including platform, base domain, and cluster name.

apiVersion: v1
baseDomain: onlinecluster.com 
compute:
- hyperthreading: Enabled   
  name: worker
  replicas: 0 
controlPlane:
  hyperthreading: Enabled   
  name: master
  replicas: 3 
metadata:
  name: ooc 
platform:
  vsphere:
    vcenter: "https://*****" 
    username: "********"
    password: "******** "
    datacenter: "<datacenter name>"
    defaultDatastore: "default data store name"
    folder: "/folder structure path, where vm's will be residing" 
fips: false 
pullSecret: '....' 
sshKey: '..........'

Creating the Kubernetes Manifest and Ignition Config Files

Generate the necessary Kubernetes manifest and Ignition config files:

openshift-install create manifests --dir=.

Open the /manifests/cluster-scheduler-02-config.yml file. locate the mastersSchedulable parameter and ensure that it is set to false. Save and exit.

rm -f openshift/99_openshift-cluster-api_master-machines-.yaml openshift/99_openshift-cluster-api_worker-machineset-.yaml

openshift-install create ignition-configs --dir=.

Creating a Web Server on the same Installation VM/Different VM

Update Your System
The first step is to update your package index. Open a terminal and run the following command:

sudo apt update

Install Apache:

sudo apt install apache2

After installation completes, Apache should start automatically. To verify:

sudo systemctl status apache2

Copy the ignitions files to Web Server

 sudo cp *.ign /var/www/html/
 ls -ltr /var/www/html/
 sudo chmod 775  /var/www/html/*.ign

Put the rhcos-4.14.0-x86_64-metal.x86_64.raw.gz in side the /var/www/html/

From the installation host, validate that the Ignition config files are available on the URLs. The following example gets the Ignition config file for the bootstrap node:

curl -k http://<HTTP_server>/bootstrap.ign

Boot ISO Image Preparation

Step 1: Boot ISO Image Using coreos-iso-maker

Download coreos-iso-maker: Use the provided GitHub repository link to download and set up coreos-iso-maker for generating the custom ISO image.

git clone https://github.com/chuckersjp/coreos-iso-maker.git
cd coreos-iso-maker

Modify inventory.yaml:

Navigate to the coreos-iso-maker directory and modify inventory.yaml as shown. Update IP addresses, network settings, and other configurations as needed.

# Example snippet from inventory.yaml
all:
  children:
    bootstrap:
      hosts:
        oocbs01.openshift.onlinecluster.com:6443:
          ipv4: 174.168.95.70
    master:
      hosts:
        dcajsbcn01.openshift.onlinecluster.com:
          ipv4: 174.168.95.74
        dcajsbcn02.openshift.onlinecluster.com:
          ipv4: 174.168.95.75
        dcajsbcn03.openshift.onlinecluster.com:
          ipv4: 174.168.95.76
    worker:
      hosts:
        oocwn01.openshift.onlinecluster.com:
          ipv4: 174.168.95.80
        oocwn02.openshift.onlinecluster.com:
          ipv4: 174.168.95.81
        oocwn03.openshift.onlinecluster.com:
          ipv4: 174.168.95.82
        oocwn04.openshift.onlinecluster.com:
          ipv4: 174.168.95.83
        oocwn05.openshift.onlinecluster.com:
          ipv4: 174.168.95.84
        oocinfra01.openshift.onlinecluster.com:
          ipv4: 174.168.95.77
        oocinfra02.openshift.onlinecluster.com:
          ipv4: 174.168.95.78
        oocinfra03.openshift.onlinecluster.com:
          ipv4: 174.168.95.79

Modify group_vars/all.yml: Update the network and other configurations in the group_vars/all.yml file.

# group_vars/all.yml
---
# If only one network interface
gateway: 174.168.95.1
netmask: 255.255.255.0
# VMWare default ens192
# KVM default ens3
# Libvirt default enp1s0
# Intel NUC default eno1
interface: ens192

dns:
  - 174.168.1.1
  - 174.168.7.7

webserver_url: <current installer VM IP>
webserver_port: 80
# Ignition subpath in http server (optional, defaults to nothing)
#webserver_ignition_path: http://192.168.66.12/master.ign
# Path to download master ignition file will be
# http://192.168.1.20:8080/ignition/master.ign

# Drive to install RHCOS
# Libvirt - can be vda
install_drive: sda

# Timeout for selection menu during first boot
# '-1' for infinite timeout. Default '10'
boot_timeout: 10

# Choose the binary architecture
# x86_64 or ppc64le
arch: "x86_64"

ocp_version: 4.14.0
iso_checksum: d15bd7ae942573eece34ba9c59e110e360f15608f36e9b83ab9f2372d235bef2
iso_checksum_ppc64: ff3ef20a0c4c29022f52ad932278b9040739dc48f4062411b5a3255af863c95e
iso_name: rhcos-{{ ocp_version }}-x86_64-live.x86_64.iso
iso_name_ppc64: rhcos-{{ ocp_version }}-ppc64le-installer.ppc64le.iso
rhcos_bios: rhcos-{{ ocp_version }}-x86_64-metal.x86_64.raw.gz

Step 2: Ansible Playbook Execution

Run Ansible playbook: Execute the Ansible playbook (playbook-single.yml) to automate the provisioning of OpenShift nodes.

#Execute Ansible playbook
ansible-playbook playbook-single.yml -K

Step 3: Uploading and Provisioning VMs

Upload ISO for Bootstrap Node:
Upload the custom ISO to the VM environment. Ensure disk.enableUUID attribute is set to true in advanced settings.
Provision Bootstrap Node: Wait for the bootstrap node to complete installation until the login screen appears.
Continue with Master and Worker Nodes: Follow the same process to upload and provision first the master nodes and then the worker nodes.

Step 4: Monitoring and Verification

Monitor Bootstrap Process:

Check the progress of the bootstrap process using the OpenShift installer command.

# Monitor bootstrap process
./openshift-install --dir <installation_directory> wait-for bootstrap-complete --log-level=info

Verify Node Status: Confirm that all nodes are ready by checking their status.

# Verify node status
oc get nodes

Logging in to the cluster by using the CLI

You installed the oc CLI.
Export the kubeadmin credentials:
export KUBECONFIG=/auth/kubeconfig
For , specify the path to the directory that you stored the installation files in.
oc whoami
sample output : system:admin

To approve all pending CSRs, run the following command:

oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

oc get nodes

DEV Community: Anzal Beg