DEV Community

Ana Villar
Ana Villar

Posted on

Building an OpenShift 4.18 Cluster from Scratch: Part 3 - The Deployment Lifecycle & Troubleshooting

#tutorial #openshift #redhat

Note: This is Part 3 of a multi-part series.

✅ [Part 1: The Network Foundation & Utilities Server] (Published)
✅ [Part 2: Generating Ignition Configs & VM Prep] (Published)
Part 4: Post-Install Hardening & User Management

In Parts 1 and 2, we built the foundation and prepared our Ignition configurations. The utilities server was humming, DHCP and DNS were routing requests correctly, and our VMs were ready to PXE boot. Now comes the moment of truth: actually deploying the cluster.

But here's the thing about bare metal OpenShift deployments—they rarely go perfectly on the first try. In this part, I'll walk through exactly what happened when my first deployment attempt failed, how I recovered, and the critical troubleshooting steps every OpenShift administrator should know.

Review we have all in place

At this point I want you to check that everything is ready to go.

  • You have created 7 virtual machines, with the specifications indicated in Part 1, under Lab Environment Overview section.
  • The VMs are called bootstos, masteros01, masteros02, masteros03, workeros01, workeros02 and workeros03.
  • Those have been created with UEFI (or EFI) firmware.
  • You have disabled Secured Boot, as explained in Part 2, under the section Disabling Secure Boot on VMs.
  • And you have noted the MAC Address of the NIC attached.

Then, ensure you have updated the Kea configuration file (/etc/kea/kea-dhcp4.conf) in your utilities server VM with the correct MAC address (hw-address), under reservations.

"reservations": [
          { "hw-address": "AA:BB:CC:DD:EE:01", "ip-address": "192.168.110.100", "hostname": "bootstos.oc41827.internal.local" },

          { "hw-address": "AA:BB:CC:DD:EE:02", "ip-address": "192.168.110.101", "hostname": "masteros01.oc41827.internal.local" },
...
Enter fullscreen mode Exit fullscreen mode

If you have to update the file now, check the syntax using

$ sudo kea-dhcp4 -t /etc/kea/kea-dhcp4.conf
Enter fullscreen mode Exit fullscreen mode

and restart the service.

$ sudo systemctl restart kea-dhcp4
Enter fullscreen mode Exit fullscreen mode

Start the deployment

Boot the 7 virtual machines

Using your Cockpit interface or with the command sudo virsh start (or, if using a different hypervisor, whatever CLI or GUI interface you prefer), start the virtual machines.

Start first with bootstos. When the iPXE menu appears, ensure you select the menu entry called Install OpenShift 4.18.27 Bootstrap. In a few moments, the process will complete and it will automatically reboot.

If, for whatever reason, you had to manually force the VM to boot from network, once the installation is completed, shutdown the VM and change the boot order to boot from its local disk.

Do the same for the other six virtual machines.

  • For masteros01, masteros02 and masteros03, when the iPXE menu appears, select Install OpenShift 4.18.27 Master Node.
  • For workeros01, workeros02 and workeros03, when the iPXE menu appears, select Install OpenShift 4.18.27 Worker Node.

Launch and monitor the progress

Now that the 7 VMs are up and running, open two SSH sessions to your utilities server, one for the actual deployment and the second to be able to monitor the progress.

In the first session, start the deployment running the following command:

$ openshift-install --dir=./oc41827upi wait-for bootstrap-complete --log-level debug
Enter fullscreen mode Exit fullscreen mode

In the second session, connect to the bootstrap VM:

$ ssh -i .ssh/oc41827upi core@bootstos
Enter fullscreen mode Exit fullscreen mode

and once there, monitor the deployment from here as well:

$ sudo journalctl -b -f -u release-image.service -u bootkube.service
Enter fullscreen mode Exit fullscreen mode

🚨 Certificate Expiration

If you followed the steps the ignition files (getting the pull secret and updating the install-config.yaml configuration file, as indicated in the section Generating installation configuration file) a few days back, you are most likely to see the deployment failing after a few minutes, and errors similar to those in the bootstrap ssh session:

May 01 13:34:49 bootstos.oc41827.internal.local bootkube.sh[4018]: [#3964] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
May 01 13:34:50 bootstos.oc41827.internal.local cluster-bootstrap[4049]: [#3965] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-05-01T13:34:50Z is after 2026-04-27T18:15:56Z
May 01 13:34:50 bootstos.oc41827.internal.local bootkube.sh[4018]: [#3965] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-05-01T13:34:50Z is after 2026-04-27T18:15:56Z
Enter fullscreen mode Exit fullscreen mode

What Happened? OpenShift certificates generated during the install-config.yaml creation have a limited validity window (typically 24 hours). My first deployment attempt came four days after configuration generation, causing the certificates to expire before installation began.

The Solution: Regenerate Everything, I am afraid.

As seen in Part 2, those are the steps needed:

  • Get a fresh pull secret from console.redhat.com
  • Move old config fiels aside
  • Recreate the configuration directory: 
$ mv oc41827upi oc41827upi.old
$ mkdir oc41827upi
$ cd oc41827upi
Enter fullscreen mode Exit fullscreen mode
  • Copy the fresh pull secret** and format it properly: 
$ python3 -m json.tool pull-secret > pull-secret.json
$ cat pull-secret.json | jq . -c > ~/oc41827upi/pull-secret-online.json
Enter fullscreen mode Exit fullscreen mode
  • Update install-config.yaml 
$ cat <<EOF > ~/oc41827upi/install-config.yaml
apiVersion: v1
baseDomain: internal.local
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 3
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: oc41827
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
fips: false
pullSecret: '$(cat ~/oc41827upi/pull-secret.json)'
sshKey: '$(cat ~/.ssh/oc41827upi.pub)'
EOF
Enter fullscreen mode Exit fullscreen mode
  • Regenerate manifests and Ignition files: 
$ cd ~/oc41827upi
$ openshift-install create manifests --dir=.
$ openshift-install create ignition-configs --dir=.
Enter fullscreen mode Exit fullscreen mode
  • Copy new Ignition files to Apache: 
$ sudo cp *.ign /var/www/html/oc41827/ignition/
$ sudo chmod +r /var/www/html/oc41827/ignition/*
$ sudo restorecon -Rv /var/www/html
Enter fullscreen mode Exit fullscreen mode
  • With fresh configurations, reset the VMs:
  1. Shut down all nodes
  2. Delete their virtual disks and attach a new one (to force clean installs)
  3. Boot via iPXE menu selection

💡 Key Lesson: Keep your configuration workflow tight. Don't generate Ignition files until you're ready to deploy immediately afterward.

Bootstrap Complete: Removing the Temporary Node

If all goes well, you can expect outputs similar to this on your utilities ssh session:

$ openshift-install --dir=./oc41827upi wait-for bootstrap-complete --log-level debug
DEBUG OpenShift Installer 4.18.37  
...
INFO Bootstrap etcd member has been removed       
INFO It is now safe to remove the bootstrap resources 
DEBUG Time elapsed per stage:                      
DEBUG Bootstrap Complete: 26m48s                   
INFO Time elapsed: 26m48s                         
$
Enter fullscreen mode Exit fullscreen mode

and this on the bootstrap session:

[core@bootstos ~]$ sudo journalctl -b -f -u release-image.service -u bootkube.service
May 01 13:55:00 bootstos.oc41827.internal.local cluster-bootstrap[4041]: "99_openshift-machineconfig_99-master-ssh.yaml": failed to create machineconfigs.v1.machineconfiguration.openshift.io/99-master-ssh -n : the server could not find the requested resource
May 01 13:55:00 bootstos.oc41827.internal.local cluster-bootstrap[4041]: "99_openshift-machineconfig_99-worker-ssh.yaml": failed to create machineconfigs.v1.machineconfiguration.openshift.io/99-worker-ssh -n : the server could not find the requested resource
May 01 13:55:00 bootstos.oc41827.internal.local cluster-bootstrap[4041]: Created "0000_00_cluster-version-operator_01_adminack_configmap.yaml" configmaps.v1./admin-acks -n openshift-config
...
May 01 14:23:13 bootstos.oc41827.internal.local bootkube.sh[5815]: Success while trying to reach API_INT_URL's https endpoint at https://api-int.oc41827.internal.local:6443/readyz
May 01 14:23:13 bootstos.oc41827.internal.local bootkube.sh[5815]: bootkube.service complete
May 01 14:23:13 bootstos.oc41827.internal.local systemd[1]: bootkube.service: Deactivated successfully.
May 01 14:23:13 bootstos.oc41827.internal.local systemd[1]: bootkube.service: Consumed 4.396s CPU time.
Enter fullscreen mode Exit fullscreen mode

This typically takes 25-40 minutes depending on your hardware and network speed.

Once bootstrap-complete succeeds, OpenShift automatically removes the bootstrap etcd member. Now we must manually remove the bootstrap node from our HAProxy load balancer:

Edit HAProxy Configuration 

$ sudo vim /etc/haproxy/haproxy.cfg
Enter fullscreen mode Exit fullscreen mode

Comment out or remove the bootstrap server entries from both the API server and machine config server listen sections:

# round robin balancing for RHOCP Kubernetes API Server
#server bootstos bootstos.oc41827.internal.local:6443 verify none check check-ssl inter 10s fall 2 rise 3 backup

# round robin balancing for RHOCP Machine Config Server
#server bootstos bootstos.oc41827.internal.local:22623 check inter 1s backup
Enter fullscreen mode Exit fullscreen mode

Then restart HAProxy:

$ sudo systemctl restart haproxy
Enter fullscreen mode Exit fullscreen mode

Shutdown bootstrap

Now you can shutdown the bootstrap machine. Using the session you opened to monitor the progress, press Crtl+C to finish the journalctl command and use the shutdown command.

[core@bootstos ~]$ sudo shutdown -h now
[core@bootstos ~]$ Connection to bootstos closed by remote host.
Connection to bootstos closed.
$
Enter fullscreen mode Exit fullscreen mode

Finish the deployment

Check the status of the OpenShift cluster on your ssh session to the utilities server.

$ export KUBECONFIG=~/oc41827upi/auth/kubeconfig
$ oc whoami
system:admin
$ oc get nodes
NAME                                      STATUS   ROLES                  AGE   VERSION
masteros01.oc41827.internal.local   Ready    control-plane,master   32m   v1.31.14
masteros02.oc41827.internal.local   Ready    control-plane,master   30m   v1.31.14
masteros03.oc41827.internal.local   Ready    control-plane,master   30m   v1.31.14
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          36m     Unable to apply 4.18.37: some cluster operators are not available
$
Enter fullscreen mode Exit fullscreen mode

Where are the worker nodes? Wait, we haven't finish yet, so let's run this:

$ openshift-install --dir=./oc41827upi wait-for install-complete --log-level=debug
Enter fullscreen mode Exit fullscreen mode

On another session, keep an eye on Certificate Signing Requests (CSRs) being pending for approval:

$ export KUBECONFIG=~/oc41827upi/auth/kubeconfig
$ oc get csr
Enter fullscreen mode Exit fullscreen mode

You might see an output similar to this:

NAME                                             AGE   SIGNERNAME                                    REQUESTOR                                                                         REQUESTEDDURATION   CONDITION
csr-2zppd                                        35m   kubernetes.io/kubelet-serving                 system:node:masteros02.oc41827.internal.local                                     <none>              Approved,Issued
...
csr-hnt8m                                        25m   kubernetes.io/kube-apiserver-client           system:node:masteros03.oc41827.internal.local                                     24h                 Approved,Issued
csr-lldhf                                        1s    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Pending
csr-lrphg                                        35m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-pqbm8                                        25m   kubernetes.io/kube-apiserver-client           system:node:masteros02.oc41827.internal.local                                     24h                 Approved,Issued
csr-v8wtg                                        35m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Approved,Issued
csr-xwtxx                                        26m   kubernetes.io/kube-apiserver-client           system:node:masteros02.oc41827.internal.local                                     24h                 Approved,Issued
...
Enter fullscreen mode Exit fullscreen mode

Those pending CSR need to be approved. This is where many administrators get stuck. When nodes first join the cluster, they request certificates for communication with the API server. Until these are approved, nodes remain in NotReady state.

Approving Certificate Signing Requests (CSRs)

Manual Approval

Approve pending CSRs individually:

$ oc adm certificate approve csr-lldhf
Enter fullscreen mode Exit fullscreen mode

Automated Batch Approval
For efficiency, especially when workers join, use this command to approve all pending CSRs at once:

$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
Enter fullscreen mode Exit fullscreen mode

Keep reviewing the list of CSRs and keep approving until the deployment has finished.

Output confirming deployment success 

$ openshift-install --dir=oc41827upi wait-for install-complete --log-level=debug
DEBUG OpenShift Installer 4.18.37                  
DEBUG Built from commit 488926dc2c95d96460ee9939929a76ed23e1c596 
...
INFO All cluster operators have completed progressing 
INFO Checking to see if there is a route at openshift-console/console... 
DEBUG Route found in openshift-console namespace: console 
DEBUG OpenShift console route is admitted          
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 
INFO     export KUBECONFIG=/home/labadmin/oc41827upi/auth/kubeconfig 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.oc41827.internal.local 
INFO Login to the console with user: "kubeadmin", and password: "ABcde-fghIKk-lmnpQ-RSTUv" 
DEBUG Time elapsed per stage:                      
DEBUG Cluster Operators Available: 2m23s           
INFO Time elapsed: 2m23s                          
$
Enter fullscreen mode Exit fullscreen mode

⚠️ Security Warning: Save that kubeadmin password somewhere secure — you should only use it to create permanent admin users and for very specific tasks.

✅ Verifying Cluster Health

Once the installer reports completion, verify everything is working:

All Nodes Ready

$ oc get nodes
Enter fullscreen mode Exit fullscreen mode

Expected output:

NAME                                      STATUS   ROLES                  AGE    VERSION
masteros01.oc41827.internal.local   Ready    control-plane,master   134m   v1.31.14
masteros02.oc41827.internal.local   Ready    control-plane,master   133m   v1.31.14
masteros03.oc41827.internal.local   Ready    control-plane,master   132m   v1.31.14
workeros01.oc41827.internal.local   Ready    worker                 96m    v1.31.14
workeros02.oc41827.internal.local   Ready    worker                 96m    v1.31.14
workeros03.oc41827.internal.local   Ready    worker                 96m    v1.31.14
Enter fullscreen mode Exit fullscreen mode

No Degraded Operators

$ oc get clusteroperator
Enter fullscreen mode Exit fullscreen mode

Every row should show True in the Available column, False in Progressing, and False in Degraded.

NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.18.37   True        False         False      28m     
baremetal                                  4.18.37   True        False         False      120m 
...
service-ca                                 4.18.37   True        False         False      121m    
storage                                    4.18.37   True        False         False      121m
Enter fullscreen mode Exit fullscreen mode

Cluster seen as available

$ oc get clusterversion
Enter fullscreen mode Exit fullscreen mode 
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.18.37   True        False         25m     Cluster version is 4.18.37
Enter fullscreen mode Exit fullscreen mode

🧪 Testing Access: The Web Console

The web console is only accessible from within your lab's network unless you configure external routing. On my local workstation, I added entries to /etc/hosts:

192.168.110.22 console-openshift-console.apps.oc41827.internal.local
192.168.110.22 oauth-openshift.apps.oc41827.internal.local
192.168.110.21 api.oc41827.internal.local
Enter fullscreen mode Exit fullscreen mode

Navigate to https://console-openshift-console.apps.oc41827.internal.local and log in with the kubeadmin credentials shown above.

🛡️ Remember: The kubeadmin account should be disabled or replaced ASAP. Create proper administrative users instead (we'll cover this in Part 4).

🏃 Stopping and Restarting the Cluster

⚠️ Critical: Always shut down workers before control plane nodes. Shutting down all etcd members simultaneously can cause quorum loss and data corruption.

Because this is a VM-based lab, I periodically shut down and restarted the cluster. Here's the correct order:

Shutdown Order

  1. Worker nodes (in any order): 
$ ssh -i .ssh/oc41827upi core@workeros03
The authenticity of host 'workeros03 (192.168.110.113)' can't be established.
...
---
[core@workeros03 ~]$ sudo shutdown -h now
[core@workeros03 ~]$ Connection to workeros03 closed by remote host.
Connection to workeros03 closed.
$
Enter fullscreen mode Exit fullscreen mode 
$ ssh -i .ssh/oc41827upi core@workeros02
...
---
[core@workeros02 ~]$ sudo shutdown -h now
[core@workeros02 ~]$ Connection to workeros02 closed by remote host.
Connection to workeros02 closed.
$
Enter fullscreen mode Exit fullscreen mode 
$ ssh -i .ssh/oc41827upi core@workeros01
...
---
[core@workeros01 ~]$ sudo shutdown -h now
[core@workeros01 ~]$ Connection to workeros01 closed by remote host.
Connection to workeros01 closed.
$
Enter fullscreen mode Exit fullscreen mode
  1. Control plane nodes (in any order): 
$ ssh -i .ssh/oc41827upi core@masteros03
...
---
[core@masteros03 ~]$ sudo shutdown -h now
[core@masteros03 ~]$ Connection to masteros03 closed by remote host.
Connection to masteros03 closed.
$
Enter fullscreen mode Exit fullscreen mode 
$ ssh -i .ssh/oc41827upi core@masteros02
...
---
[core@masteros02 ~]$ sudo shutdown -h now
[core@masteros02 ~]$ Connection to masteros02 closed by remote host.
Connection to masteros02 closed.
$
Enter fullscreen mode Exit fullscreen mode 
$ ssh -i .ssh/oc41827upi core@masteros01
...
---
[core@masteros01 ~]$ sudo shutdown -h now
[core@masteros01 ~]$ Connection to masteros01 closed by remote host.
Connection to masteros01 closed.
$
Enter fullscreen mode Exit fullscreen mode

Startup Order

  1. Control plane nodes (wait for them to stabilize)
  2. Worker nodes (wait for CSR approvals)

After restarting, you might see pending CSRs. Approve them using the batch command from earlier, and nodes will return to Ready status within 5-10 minutes.

📋 Summary: Deployment Checklist

✅ Pull secret downloaded
✅ Ignition files regenerated and served
✅ VMs PXE booted from iPXE menu
✅ Deployment started
✅ Bootstrap monitored via journalctl
✅ Bootstrap removed from HAProxy
✅ CSRs approved
✅ Deployment finished
✅ Cluster operators verified stable

🔜 Coming Up in Part 4

Now that the cluster is running, we need to make it usable:

  • Creating HTPasswd users (admin and developer accounts)
  • Assigning roles (cluster-admin vs project-specific permissions)
  • Testing logins with new credentials
  • Disabling the kubeadmin account (security best practice)

The cluster is alive—but until we configure proper authentication, you're still flying blind with the emergency kubeadmin account. Stay tuned!

Did your deployment run smoothly, or did you hit certificate expiration issues too? Share your experiences in the comments below! 👇

Top comments (0)