DEV Community

Ana Villar
Ana Villar

Posted on

Building an OpenShift 4.18 Cluster from Scratch: Part 4 - Post-Install Hardening & User Management

#openshif #tutorial #htpasswd #redhat

Note: This is Part 4 (the final instalment!) of a multi-part series.

Part 1: The Network Foundation & Utilities Server
Part 2: Generating Ignition Configs & VM Prep
Part 3: The Deployment Lifecycle & Troubleshooting
Part 4: Post-Install Hardening & User Management (You are here)

The cluster is up. All six nodes report Ready, every cluster operator is stable, and the web console loads without errors. You could stop here—but you shouldn't. Right now, your only administrative access is through the temporary kubeadmin account, and that's not a sustainable way to manage a production-like environment.

In this final part, we'll lock things down: configure proper authentication, assign roles, verify access levels, and establish safe procedures for shutting down and restarting the cluster.

🔑 Why Hardening Matters

The kubeadmin account is a bootstrap credential auto-generated during installation. It has unrestricted cluster-admin privileges and its password is displayed in plaintext during the install. In a real environment, this account should be removed as soon as proper identity providers are configured.

Even in a lab, practicing proper user management ensures you understand:

  • How OpenShift's OAuth system works
  • How to map external identity providers to cluster roles
  • How RBAC controls what each user can do

👤 Creating HTPasswd Users

OpenShift supports several identity providers (LDAP, OIDC, GitHub, etc.). For a lab environment, HTPasswd is the simplest option—it uses a flat file with bcrypt-hashed passwords.

Install httpd-tools

You'll need the htpasswd utility, which comes from the httpd-tools package:

$ sudo dnf install httpd-tools
Enter fullscreen mode Exit fullscreen mode

Generate the HTPasswd File

Create a working directory and generate bcrypt-hashed credentials:

$ mkdir ~/openshift && cd ~/openshift
Enter fullscreen mode Exit fullscreen mode

Create the file with the first user (-c creates, -B forces bcrypt)

$ htpasswd -c -B htpasswd admin
Enter fullscreen mode Exit fullscreen mode

Add additional users (no -c, to avoid overwriting)

$ htpasswd -B htpasswd developer
Enter fullscreen mode Exit fullscreen mode

You'll be prompted for passwords interactively.

Create the Kubernetes Secret

OpenShift's OAuth operator reads the HTPasswd file from a Kubernetes secret in the openshift-config namespace:

$ oc create secret generic localusers --from-file htpasswd=/home/labuser/openshift/htpasswd -n openshift-config
Enter fullscreen mode Exit fullscreen mode

Verify it was created:

$ oc get secret localusers -n openshift-config
Enter fullscreen mode Exit fullscreen mode

⚙️ Configuring the OAuth Identity Provider

Now we need to tell OpenShift to use this HTPasswd file as an identity provider.

Via the Web Console

The easiest way to configure OAuth for the first time is through the web console:

Navigate to Administration → Cluster Settings → Configuration. Click on the OAuth resource.

Under Identity providers, unfold Add, and select HTPasswd

Give it a name, for example, localusers, and browse to the location of the htpasswd file previously created.

Press Add

Using command line

Optionally, you can edit the OAuth configuration directly using the YALM tab, or you can export and do the changes manually:

$ oc get oauth cluster -o yaml > oauth.yaml
Enter fullscreen mode Exit fullscreen mode

Edit the oauth.yaml file to add the identity provider under spec:

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
...
spec: {}
Enter fullscreen mode Exit fullscreen mode 
apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
...
spec:
  identityProviders:
  - htpasswd:
      fileData:
        name: localusers
    mappingMethod: claim
    name: my_localusers
    type: HTPasswd
Enter fullscreen mode Exit fullscreen mode

And then apply the changes:

$ oc apply -f oauth.yaml
Enter fullscreen mode Exit fullscreen mode

What Happens Next

Authentication changes require redeploying pods in the openshift-authentication namespace. The oauth-openshift pods restart and the new pods are running.

$ oc get po -n openshift-authentication
Enter fullscreen mode Exit fullscreen mode 
NAME                              READY   STATUS    RESTARTS   AGE
oauth-openshift-b5b5465df-r8zwh   1/1     Running   0          5m30s
oauth-openshift-b5b5465df-tz48h   1/1     Running   0          5m11s
oauth-openshift-b5b5465df-v9m7f   1/1     Running   0          4m44s
Enter fullscreen mode Exit fullscreen mode

The new three pods have been running since we added the new identity provider.

🎖️ Assigning Roles

By default, new users have no permissions—they can authenticate but can't do anything. We need to grant the admin user cluster-admin privileges while leaving developer with standard (restricted) access.

Grant Cluster-Admin to admin

Let´s start verifying existing bindings for administration accounts:

$ oc get clusterrolebinding | grep ^cluster-admin
Enter fullscreen mode Exit fullscreen mode

We can expect an output similar to this:

cluster-admin                                                               ClusterRole/cluster-admin                                                               36d
cluster-admins                                                              ClusterRole/cluster-admin                                                               36d
Enter fullscreen mode Exit fullscreen mode

To assign the administration role to the new admin account:

$ oc adm policy add-cluster-role-to-user cluster-admin admin
Enter fullscreen mode Exit fullscreen mode

If we need to create a project-specific admin:

$ oc adm policy add-role-to-user admin <username> -n <project_name>
Enter fullscreen mode Exit fullscreen mode

💡RBAC Design: In a production environment, you'd typically use more granular roles—granting cluster-admin only to break-glass accounts and using project-scoped roles for daily administration. For this lab, the two-tier model (admin vs developer) demonstrates the principle clearly.

🧪 Testing User Access

Admin Login 

$ oc login -u admin -p xxxxxxxxx
Enter fullscreen mode Exit fullscreen mode

(Replace xxxxxxxxx with the actual password)

Verify admin can see cluster resources:

$ oc get nodes
NAME                                            STATUS   ROLES           AGE   VERSION
masteros01.oc41827.internal.local               Ready    control-plane   36d   v1.31.14
masteros02.oc41827.internal.local               Ready    control-plane   36d   v1.31.14
masteros03.oc41827.internal.local               Ready    control-plane   36d   v1.31.14
workeros01.oc41827.internal.local               Ready    worker          36d   v1.31.14
workeros02.oc41827.internal.local               Ready    worker          36d   v1.31.14
workeros03.oc41827.internal.local               Ready    worker          36d   v1.31.14
Enter fullscreen mode Exit fullscreen mode

Developer Login 

$ oc login -u developer -p yyyyyyy
Enter fullscreen mode Exit fullscreen mode

(Replace yyyyyyy with the actual password)

Verify developer is properly restricted:

$ oc get nodes
Error from server (Forbidden): nodes is forbidden: User "developer" cannot list resource "nodes" in API group "" at the cluster scope
$
Enter fullscreen mode Exit fullscreen mode

This confirms RBAC is working correctly. The developer can authenticate but cannot access cluster-scoped resources.

Verify Identities Were Created

After logging in as both users, check that identities were properly recorded:

$ oc login -u admin -p xxxxxxxxx
$ oc get users
NAME       UID                                    IDENTITIES
admin      eb84cf5d-...........................   localusers:admin
developer  e62f33b7-...........................   localusers:developer
$ oc get identity
NAME                    IDP NAME     IDP USER NAME   USER NAME   USER UID
localusers:admin        localusers   admin           admin       eb84cf5d-...
localusers:developer    localusers   developer       developer   e62f33b7-...
$
Enter fullscreen mode Exit fullscreen mode

🛡️ Certificate Validity

Before wrapping up, it's wise to check when your cluster certificates expire.

$ oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.metadata.annotations.auth\.openshift\.io/certificate-not-after}'
2027-05-01T13:43:53Z
$
Enter fullscreen mode Exit fullscreen mode

Certificates are valid for approximately one year from installation. The cluster's certificate rotation operators handle renewal automatically, but it's good practice to verify—especially in long-running labs.

🖥️ Accessing the Web Console

From your workstation, navigate to the OpenShift console web page.

Log in with either admin or developer.

The admin sees the full Administrator perspective, while the developer gets the Developer perspective with project-scoped views.

🗑️ Removing the kubeadmin Account

Now that we have a working admin user with cluster-admin privileges, the temporary kubeadmin account has outlived its purpose. But before we remove it, let's understand what it is, why it should go, and what our options are.

What Is kubeadmin?

During installation, OpenShift auto-generates a break-glass account called kubeadmin. It has unrestricted cluster-admin privileges (essentially the root user of the cluster). The password is dynamically generated and displayed in plaintext in the installer's output:

INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 
INFO     export KUBECONFIG=/home/labadmin/oc41827upi/auth/kubeconfig 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.oc41827.internal.local 
INFO Login to the console with user: "kubeadmin", and password: "........................."
Enter fullscreen mode Exit fullscreen mode

It's also stored as a bcrypt hash in a Kubernetes secret:

$ oc get secrets -n kube-system kubeadmin -o yaml 

apiVersion: v1
data:
  kubeadmin: JDJhJDEwJE85dkxPREtjYWQ4TlJiTEV6eHhKQk9OODBWaU1qbkJoeU5jQ1paOWJOa2gwS3duNzRtWkJh
...

$ oc get secrets -n kube-system kubeadmin -o jsonpath="{.data.kubeadmin}" | base64 -d
$2a$10$O9vLODKcad8NRbLEzxxJBON80ViMjnBhyNcCZZ9bNkh0Kwn74mZBa
Enter fullscreen mode Exit fullscreen mode

Note that you cannot decrypt the password because the file stores one-way cryptographic hashes rather than encrypted text. There is no online tool or mathematical method to reverse a secure hash back into the original plain-text password. You could

  • Brute-forcing: You can attempt to guess the password by hashing candidate words and comparing them to the stored hash using local tools like Hashcat.
  • Resetting: Generate a new credential using an Htpasswd Generator and replace the old entry, see below.

Why Remove It?

There are several reasons:

  • The password is exposed in plaintext. Anyone with access to the installer output, shell history, or .openshift_install.log has the full kubeadmin credentials.
  • No accountability. The kubeadmin account is shared—there's no individual identity tied to actions taken with it. Audit logs show kubeadmin, not a real person.
  • No rotation mechanism. Unlike users managed through an identity provider, kubeadmin's password can't be rotated through normal OAuth workflows. You'd have to manually patch the secret.
  • It's unnecessary. Once you have a properly configured identity provider with at least one cluster-admin user, kubeadmin serves no purpose.

How to reset the KubeAdmin password?

If you feel uneasy to delete the kubeadmin account, this Red Hat document explains how to change its password.

Using the htpasswd CLI tool, we can create a password hash string:

$ htpasswd -bnBC 10 "" bxYJn-jvZBy-viWJi-HGxZA 
:$2y$10$y9MHpYM8LWGvN.K8n7mSbuOKTeid39XRn2orARXJDFa9DzEB4o/36

$
Enter fullscreen mode Exit fullscreen mode

The htpasswd needs to be converted into base64 encoding, and the first character ':' must be removed to create a kubeadmin-compatible hash string as follows:

$ htpasswd -bnBC 10 "" bxYJn-jvZBy-viWJi-HGxZA | cut -c 2- | base64
JDJ5JDEwJGdFbjRyTGI0ZTlmcXFGOTNLZlRubnVpR01CSWgxdDVjMzY0UXVqSTZzLzVRTmJFSDFxTHBLCgo=
$
Enter fullscreen mode Exit fullscreen mode

We can now use the output from the htpasswd tool to set the secret value in OpenShift:

$ oc patch -n kube-system secret/kubeadmin --patch '{"data": {"kubeadmin": "JDJ5JDEwJGdFbjRyTGI0ZTlmcXFGOTNLZlRubnVpR01CSWgxdDVjMzY0UXVqSTZzLzVRTmJFSDFxTHBLCgo="}}'
$ oc login -u kubeadmin -p bxYJn-jvZBy-viWJi-HGxZA 
Login successful.
...
Using project "default".
$
Enter fullscreen mode Exit fullscreen mode

Prerequisites for Removal

⚠️ Critical: Red Hat's documentation is explicit on this point—if you remove kubeadmin before another user has cluster-admin privileges, you will be locked out of the cluster and the only recovery is a full reinstall. Before proceeding:

  1. You must have configured at least one identity provider.
  2. You must have added the cluster-admin role to a user.
  3. You must be logged in as an administrator.

Verify your admin user works:

$ oc login -u admin -p xxxxxxx
$ oc whoami
$ oc get nodes
Enter fullscreen mode Exit fullscreen mode

Removing kubeadmin

It's a single command:

$ oc delete secrets kubeadmin -n kube-system
Enter fullscreen mode Exit fullscreen mode

Once deleted, the kubeadmin account can no longer authenticate against the cluster. OpenShift will not automatically regenerate the secret.

💡 Lab Tip: In a lab environment, some administrators choose to keep kubeadmin but rotate the password to something memorable. In production, the best practice is to remove it entirely and rely on your identity provider—recreating it only during emergencies.

Final Verification

After removal, confirm that both your regular users still function normally:

$ oc login -u admin -p xxxxxxx
$ oc whoami    # Should return: admin
Enter fullscreen mode Exit fullscreen mode 
$ oc login -u developer -p yyyyyyyyy
$ oc whoami    # Should return: developer
Enter fullscreen mode Exit fullscreen mode

And confirm kubeadmin is truly gone:

$ oc login -u kubeadmin -p '......................'
# Should fail with: Unauthorized
Enter fullscreen mode Exit fullscreen mode

📋 Series Wrap-Up: What We Built

Over four parts, we went from an empty RHEL 10 hypervisor to a fully functional OpenShift 4.18 cluster:

Part 1: DNS, DHCP, TFTP, HAProxy, NTP, and Apache infrastructure
Part 2: OpenShift installer, Ignition configs, RHCOS images, and Secure Boot removal
Part 3: Bootstrap lifecycle, CSR approval, install completion, and cluster lifecycle
Part 4: OAuth identity provider, RBAC and user management

Key Lessons Across the Series

  • Timing matters: Ignition certificates expire—deploy promptly after generating configs
  • Firewalls: Every port OpenShift needs (6443, 22623, 80, 443) must be explicitly opened
  • CSRs are manual in UPI: The cluster won't self-heal until you approve node certificates
  • Order of operations: Proper shutdown/startup sequences prevent etcd quorum loss
  • HTPasswd is just the beginning: In production, integrate with LDAP or OIDC for centralized identity management

🔗 References

Thank you for following along this series! If you have questions, ran into different issues in your own deployment, or want to share your bare-metal OpenShift stories, drop a comment below. I'd love to hear how your lab turned out. 🚀

Top comments (0)