Master the Migration: Upgrading Your RHEL Workload - Part 1

Upgrading Red Hat Enterprise Linux 8 to 9: A Practical Guide Using Leapp

Introduction

As Linux administrators and engineers, we eventually reach a point where critical systems must be upgraded without unnecessarily disrupting business operations.

At first, upgrading a perfectly functioning operating system may seem unnecessary. After all, why disturb a system that is working as expected?

However, enterprise environments cannot rely indefinitely on systems simply because they continue to function. Security vulnerabilities, software lifecycle policies, vendor support requirements, and evolving technology standards make operating system upgrades an essential part of infrastructure management.

But upgrading RHEL is only part of the story!

In this series, I’ll go beyond the traditional upgrade approach and explore different paths to modernize your Linux infrastructure. We will go from conventional in-place upgrades and fresh installations to the emerging world of Image Mode with bootc.

What are the trade-offs? Which approach works best for different environments? And how can we minimize downtime while keeping critical workloads running? I’ll explore these questions hands-on, uncover the challenges, and share practical lessons along the way.

The journey from traditional Linux management to modern, image-based operating systems starts here. Stay tuned. You won’t want to miss what’s next!

In this article however, I will explain why upgrading from Red Hat Enterprise Linux (RHEL) 8 to RHEL 9 matters, the precautions to take before beginning the process, and how to approach an in-place upgrade using Leapp.

The objective is to share practical experience, highlight common challenges, and help Linux administrators plan and execute upgrades with greater confidence.

Why Upgrade from RHEL 8 to RHEL 9?

An operating system upgrade is more than a version change. It is an opportunity to improve security, maintain vendor supportability, and prepare infrastructure for future requirements. Here are some reasons to consider upgrading.

Improved Security and Compliance:
Security and compliance teams continually assess infrastructure for vulnerabilities, outdated software, and deviations from organizational security policies.

RHEL 9 introduces newer security capabilities, updated cryptographic policies, and security improvements that can help organizations meet their security and regulatory requirements.

However, an upgrade alone does not guarantee compliance. Administrators must also review system configurations, access controls, security policies, and application requirements after the migration.

Lifecycle Planning and Vendor Support:
Every operating system has a defined support lifecycle. As a platform approaches the end of its maintenance or support period, organizations must plan how to maintain security updates, technical support, and application compatibility.

Running an unsupported operating system can increase operational risk, particularly when it hosts business-critical applications.
Upgrading to a supported release helps organizations maintain access to the appropriate vendor updates and support services, subject to their subscription and support agreements.

Access to Newer Tools and Technologies:
Newer operating system releases provide updated system libraries, development tools, virtualization capabilities, and platform features.

RHEL 9 also introduces changes that may benefit modern application development, automation, and infrastructure management.
These improvements can help organizations prepare for evolving workloads and technology requirements. However, administrators must verify that their applications, drivers, monitoring agents, and third-party software are compatible with the target release.

Understanding the Risks Before Upgrading

Although upgrading offers several benefits, it is not an activity that should be undertaken without proper planning.

A major-version upgrade can affect application dependencies, system libraries, network configurations, security policies, and third-party software.

Before proceeding, consult the relevant application teams, software vendors, and official Red Hat documentation to confirm that the proposed upgrade is supported.

For business-critical infrastructure, the upgrade should first be tested in a development or staging environment that closely resembles production.

Understanding Leapp

Traditionally, moving between major operating system versions has often involved provisioning a new server, reinstalling applications, migrating data, and switching production workloads to the replacement system.

Although this approach remains appropriate in many environments, it can require significant planning and may introduce additional migration complexity.

Red Hat provides Leapp as a framework for supported in-place upgrades between specific RHEL major versions.

Leapp assesses the existing system, identifies potential upgrade blockers, prepares the target operating system packages, and facilitates the upgrade process through a dedicated boot environment.

The process generally involves:

• Reviewing the supported upgrade path.
• Preparing the source operating system.
• Running a pre-upgrade assessment.
• Resolving reported inhibitors.
• Executing the upgrade.
• Verifying the upgraded operating system.
• Performing post-upgrade cleanup and validation.

Leapp does not eliminate the need for backups, compatibility testing, or recovery planning. A successful upgrade depends on proper preparation and the condition of the source system.

Confirm the Supported Upgrade Path

Before running any commands, establish whether your current RHEL installation can be upgraded to the intended target release.

For a typical RHEL 8 to RHEL 9 upgrade, the source system should be on the latest supported RHEL 8 minor release, commonly RHEL 8.10, with the necessary updates installed. At the time of writing, upgrades are supported from 8.10 to 9.8

Also verify whether the upgrade is supported for your system architecture, installed packages, subscription configuration, and infrastructure platform.

Review the System Requirements

Before starting an in-place upgrade, confirm that the server meets the relevant requirements.

Operating System and Architecture

For this practical exercise, the environment is based on the x86_64 architecture.

The procedure may differ for other architectures, so consult the relevant Red Hat documentation before applying the same approach elsewhere.

Subscription and Repository Access

Ensure the system has a valid RHEL subscription or an appropriately configured repository source or Satellite.

For disconnected environments, confirm that the required installation media or local repository mirrors contain the packages needed for the upgrade.

A Red Hat Developer Subscription can be suitable for eligible development and testing environments, subject to its applicable terms. I used it for this process.

Available Disk Space

Check that sufficient space is available in the relevant filesystems, particularly:

• /var
• /var/lib/leapp
• /boot
• /
• /usr

The exact space requirements depend on the environment and upgrade configuration. Review the pre-upgrade report and official documentation rather than relying on a fixed estimate.

Console Access

Ensure that out-of-band or hypervisor console access is available.
Examples include:

• Dell iDRAC
• HPE iLO
• A hypervisor console
• A supported serial console

Console access is particularly important because the upgrade process includes rebooting the system. If networking fails during the migration, remote SSH access may no longer be available.

Verified Backup and Recovery Plan

Before proceeding, ensure that a suitable backup or recovery mechanism is available and that the recovery procedure has been tested.

A backup that has never been restored is not a fully validated recovery plan.

Pre-Upgrade Planning Checklist

A structured checklist helps reduce the risk of unexpected downtime. In as much as this is not very exhaustive, tailor this to what your organization use.

Before the Upgrade

• Test the upgrade in a development or staging environment.
• Confirm that application owners have been notified.
• Validate application compatibility with the target RHEL release.
• Obtain the necessary management and change-control approvals.
• Take a backup or snapshot and verify the recovery procedure.
• Update the source system to the latest supported RHEL 8 minor release.
• Confirm that the appropriate kernel is installed and running.
• Verify subscription status and repository accessibility.
• Review third-party repositories and packages.
• Confirm console access.
• Document current services, network settings, mount points, and application dependencies.

During the Upgrade

• Run the Leapp pre-upgrade assessment.
• Review and resolve all upgrade inhibitors.
• Review any additional warnings that could affect the workload.
• Run the upgrade only after the system is ready.
• Monitor progress through the available console.
• Keep the recovery plan accessible.

After the Upgrade

• Verify the operating system release and running kernel.
• Confirm that SELinux is operating in the intended mode.
• Validate networking, DNS, routing, and authentication.
• Verify critical system services.
• Test application functionality.
• Confirm database and storage availability.
• Revalidate monitoring, logging, and security controls.
• Review temporary files, obsolete packages, and upgrade logs.
• Update operational documentation and monitoring baselines.

Performing Backups

Once you have management approvals and the upgrade has been tested, the next step is to protect the existing environment.

The backup method should be selected according to the server's role, storage layout, and recovery requirements. Let's consider a few scenarioes:

Virtual Machines

A hypervisor snapshot may provide a convenient recovery point. However, snapshots are not a substitute for an independent backup. Follow the hypervisor vendor's guidance and ensure that application consistency requirements are addressed.

Physical Servers

Depending on the environment, recovery options may include:

• Red Hat-compatible ReaR recovery images.
• Storage or logical-volume snapshots.
• File-level backups.
• Full-system backup solutions. At a minimum, protect critical configuration files, application data, and other information required to rebuild the service.

Database Servers

Database workloads require special attention. Take an appropriate logical backup or consistent database snapshot in addition to the system-level backup. Follow the database vendor's instructions for ensuring that the backup can be restored.

Application Servers

Review the application vendor's backup and recovery guidance. Where practical, install and test the application on a separate RHEL 9 system before upgrading the production server.

This helps identify unsupported dependencies before they become an operational problem.

Prepare and Update the RHEL 8 System

Before upgrading, verify the system's current state. Start by obtaining an administrative shell:

sudo -i

Check the operating system release:

cat /etc/redhat-release

Check the running kernel:

uname -r

Review the subscription status:

subscription-manager status

Inspect enabled repositories:

dnf repolist --enabled

Review repository configuration carefully, especially if the server uses Satellite, custom repository definitions, or third-party software sources.

Update the System

After confirming that the repositories are correct and the backup is available, update the source operating system according to your organization's patching procedures.
For a standard RHEL 8 environment, the following commands can be used as a starting point:

dnf clean all
dnf update -y

Review the transaction before allowing it to proceed in a production environment.

If a reboot is required, schedule it through the normal change-control process if you have not done so already for the upgrade:

reboot

After the system returns, verify the release and running kernel again:

cat /etc/redhat-release
uname -r

Confirm that the system is healthy before continuing.

Make these changes only when they are required by the supported upgrade procedure for your environment.

Review Leapp Prerequisites

Before installing or updating Leapp, inspect the existing environment for obsolete upgrade tooling, package conflicts, and unsupported third-party dependencies.

dnf remove leapp python2-leapp snactor leapp-upgrade-el7toel8 --disableexcludes=all -y

rpm -e --nodeps leapp-deps-el8 leapp-repository-deps-el8 

Older Leapp packages left over from previous major-version migrations can cause confusion or interfere with the intended procedure hence we had to remove them.

Once the older Leapp application is removed as show in the diagram above, install the newer version from the enabled repo

dnf install -y leapp-upgrade vdo kmod-kvdo --disableexcludes=all 

This installs the Leapp binaries and capabilities to manage old vdo disks with the logical volume manager.

Also check the installed storage, virtualization, security, and application components for known compatibility restrictions.

This raise the file descriptor limit. The default value is 1024 and the upgrade process will need to open a lot of files during the upgrade. Ensure it is run on the same shell where the upgrade will be initiated.

ulimit -n 65535

Set the crypto policy to LEGACY to ensure that legacy applications work, you can change this back to default or whatever crypto policy in place in your environment after the upgrade:

update-crypto-policies --set LEGACY

Disable Firewall zone drifting. This is an old feature of the firewall on older RHEL which allows packets destined for undefined zones to end up on the default firewall zone and these packets could be accepted on these zones. This feature is not accepted on RHEL 9 or other newer versions. If not disabled, Leapp will mark it as an inhibitor to the upgrade. Disable with the command below:

sed -i s/^AllowZoneDrifting=.*/AllowZoneDrifting=no/ /etc/firewalld/firewalld.conf

Third-Party Software

Third-party software can introduce upgrade blockers. Examples include:

• Docker CE and associated packages.
• Kubernetes components.
• Custom kernel modules.
• Security agents.
• Monitoring agents.
• Backup software.
• Proprietary application dependencies.

In as much as uninstalling before and reinstalling the application after the upgrade works for most cases, it might not be necessary. Check the vendor's compatibility matrix and supported migration instructions first.

Run the Leapp Pre-Upgrade Assessment

The pre-upgrade assessment is one of the most important stages of the migration. It evaluates the system and generates a report identifying potential issues.

On a connected system using the standard Red Hat subscription and repository workflow, the assessment is generally initiated with:

leapp preupgrade

Without RHSM access, Leapp can use local repositories or ISO images:

leapp upgrade --no-rhsm --enablerepo rhel-9-baseos --enablerepo rhel-9-appstream

You will need to define those repositories under /etc/yum.repos.d/ (or supply the RHEL 9 ISO) before running the command. Red Hat Satellite is the better option for fleets, because it also centralises content views and approvals.

For environments using custom repositories, disconnected installation media, or a particular target minor release, additional configuration may be required.

Use the exact command and repository configuration documented for your environment.

After the assessment completes, in most cases like the example in the image above, you will see some inhibitors(errors), inspect the report:

less /var/log/leapp/leapp-report.txt

Review the answer file when applicable:

less /var/log/leapp/answerfile

The report may identify inhibitors, warnings, and other findings that require attention.

There are situations where there is no inhibitor and in such case, you can go ahead with the upgrade.

An inhibitor is an issue that prevents the upgrade from proceeding until it has been resolved.

Troubleshooting Common Upgrade Inhibitors

The exact issues reported by Leapp depend on the installed packages, configuration, and source system history.

The following examples illustrate common categories of problems.

A. Unanswered Questions in the Answer File

Leapp may require explicit answers to certain questions before it can proceed.
Review the answer file and identify the section associated with the reported issue.
For example, if the report identifies the VDO confirmation section as requiring an answer, the documented response may be supplied using:

leapp answer --section remove_pam_pkcs11_module_check.confirm=True

Use this example only when the corresponding question is present and its answer is appropriate for the system.

Do not copy answer-file commands blindly. Different inhibitors require different responses, and some findings require configuration changes rather than a simple confirmation.

B. Unsupported Kernel Modules

Legacy or unsupported kernel modules can prevent a migration.
Examples that may be encountered in particular environments include pata_acpi and floppy.
First, determine whether the module is loaded and whether the system actually requires it.

For example:

lsmod | grep -E 'pata_acpi|floppy'

modprobe -r pata_acpi && echo "blacklist pata_acpi" > /etc/modprobe.d/blacklist-pata_acpi.conf

modprobe -r floppy && echo "blacklist floppy" > /etc/modprobe.d/blacklist-floppy.conf

If the module is confirmed to be unnecessary and the supported remediation calls for disabling it, follow the documented procedure.
Do not unload or blacklist kernel modules indiscriminately. Doing so can affect hardware, storage, or system functionality.

C. Legacy Network Configuration

Older network scripts and custom interface configurations can cause compatibility problems. Inspect the network connections:

nmcli connection show

Review legacy network configuration files:

ls -l /etc/sysconfig/network-scripts/
mv /etc/sysconfig/network-scripts/ifcfg-lo.bak /root/

Where appropriate, NetworkManager provides a command to migrate supported connection profiles:

nmcli connection migrate "ens192"

Replace ens192 with the actual connection profile name.

Before changing a production network configuration, back up the relevant files and ensure that console access is available.
Never assume that every legacy network script should be removed. Determine which files are active and whether the reported issue applies to the system.

D. Configuration Management and Watchdog Services

Configuration-management agents, automated remediation services, and hardware watchdogs can interfere with a maintenance operation if they trigger changes or reboots unexpectedly.
Identify the services installed on the system and determine whether the upgrade procedure requires any of them to be temporarily stopped.

Coordinate this with the relevant operations teams.

Do not disable monitoring, watchdogs, or configuration-management services without a clear reason and a plan to restore them after the upgrade.

E. RPM Database Warnings

Leapp may report issues involving the RPM database or package metadata.
Before attempting a repair, inspect the reported error and confirm the recommended remediation.

If a database backup and rebuild are appropriate for the specific issue, an example of the backup procedure is:

tar -C /var/lib -czf \
  "/root/rpmdb-backup-$(date +%F-%H%M%S).tar.gz" rpm

A database rebuild should be performed only when justified by the diagnostic findings and applicable vendor guidance.

rpm --verifydb
rpm --rebuilddb

Do not rebuild the RPM database merely because an unrelated warning appears in the upgrade report.

F. Root SSH Login and Authentication Configuration

Authentication configuration can introduce compatibility or security issues during a major-version upgrade.

Review any reported findings involving SSH, PAM, or authentication modules.

Do not enable root SSH login simply to work around an upgrade problem.

If a report identifies a specific authentication configuration as incompatible, follow the documented remediation while maintaining the organization's security requirements.

G. The Newest Installed Kernel is not the Running Kernel

This happens in a situation where you have a more recent kernel installed but an older kernel is booting up. Run the following commands to verify:

uname -r
grubby --default-kernel

In a normal system, both would return the same kernel. However, when both of them are not agreeing, find out the default kernel and then sue the following commmand to fix:

grubby --set-default /boot/vmlinuz-4.18.kernel.x86_64
grubby --default-kernel

Then

reboot

If the problem is not fixed, reboot and rebuild grub. First, confirm the disk partition holding the boot device, reinstall grub on it and regenerate the grub configuration file. In the example below, I assume the /boot partition is on the /dev/sda drive.

grub2-install /dev/sda 
grub2-mkconfig -o /boot/grub2/grub.cfg

Execute the In-Place Upgrade

Once all inhibitors have been resolved, review the report again and confirm that the system is ready.

Before starting, verify that:

• The backup is available.
• The recovery procedure is understood.
• The source system is healthy.
• The target release is supported.
• The required repositories are accessible.
• The application owners are ready to validate the workload.
• Console access is available.

For a standard connected system, the upgrade is generally initiated using:

leapp upgrade

Without RHSM access, Leapp can use local repositories or ISO images:

leapp upgrade --no-rhsm --enablerepo rhel-9-baseos --enablerepo rhel-9-appstream

You will need to define those repositories under /etc/yum.repos.d/ (or supply the RHEL 9 ISO) before running the command. Red Hat Satellite is the better option for fleets, because it also centralises content views and approvals.

Use the target-version and repository options required by the specific supported upgrade procedure.

For example, custom repository configurations may require additional parameters. Do not reuse a command copied from another environment without confirming that its target version and repositories match your system.

Monitor the process and wait for the expected completion state before proceeding.

What Happens During the Upgrade?

The upgrade process involves several stages.

1. Transaction preparation
Leapp evaluates the required packages and prepares the changes necessary to transition the system to the target release.

2. Upgrade environment preparation
The process prepares a dedicated boot environment for performing the migration.

3. Package migration
During the upgrade, the system replaces or updates packages and components as required by the target operating system.

4. Reboot and finalization
The system reboots into the upgrade environment and performs the migration. It subsequently boots into the upgraded operating system if the process completes successfully.

The exact behavior and duration depend on the system, installed packages, and repository configuration.

Avoid interrupting the process unless required by an established recovery procedure.

Troubleshooting an Upgrade That Does Not Complete

If the system fails to complete the upgrade, begin by examining the logs rather than immediately attempting another upgrade.
Depending on the failure stage, useful locations may include:

less /var/log/leapp/leapp-report.txt
less /var/log/leapp/leapp-upgrade.log
less /run/initramfs/rdsosreport.txt

The files available and their relevance depend on how far the upgrade progressed. For a system that boots successfully, inspect the system journal:

journalctl -b

For earlier boot attempts, review the relevant previous boot where journal history is available:

journalctl --list-boots

Then inspect the appropriate boot:

journalctl -b -1

Use the error messages to determine whether the problem involves package dependencies, boot configuration, storage, networking, or another component.

If the server cannot boot normally, use the available console and follow the recovery procedure appropriate to the failure stage.

Do not assume that a failed upgrade can always be reversed with a simple rollback command. Recovery may require restoring a validated backup or following the documented recovery procedure for the environment.

Verify the Upgraded System

After the system boots successfully, verify the operating system and confirm that the migration completed as expected.

Check the Operating System Release

cat /etc/os-release

You can also run:

cat /etc/redhat-release

Confirm that the system reports the intended RHEL 9 release.

Check the Running Kernel

uname -r

Confirm that the running kernel is appropriate for the upgraded operating system.

Check System Services

Review failed services:

systemctl --failed

Inspect the system journal for relevant errors:

journalctl -p err -b

Investigate any failures affecting the operating system or business applications.

Verify Network Connectivity

Check the network configuration:

ip address
ip route

Confirm that DNS resolution and connectivity to required services are working.
For example:

getent hosts labs.ifeatu.africa

Replace labs.ifeatu.africa with a hostname relevant to your environment.

Validate Applications

Work with application owners to confirm that:

• Critical services start successfully.
• Databases are accessible.
• Application endpoints respond correctly.
• Authentication works.
• Scheduled jobs operate as expected.
• Monitoring and alerting are functioning.
• Backup jobs remain operational.
• Performance is within acceptable limits.

A successful operating system boot is not sufficient evidence that the entire business service is healthy.

Post-Upgrade Cleanup & Operations

Once the upgraded system has been validated, review any remaining cleanup tasks.

Clean up all RHEL 8 Kernels after confirming the booted RHEL 9 Kernel is active:

uname -r
rpm -qa 'kernel*' | grep '\.el8'
dnf remove $(rpm -qa 'kernel*' | grep '\.el8')

Clean up other leftovers from the previous installations - in this case, I will be removing items from old RHEL 7 and RHEL 8 installations:

rpm -qa | grep -e '\.el[78]' | grep -vE '^(gpg-pubkey|libmodulemd|katello-ca-consumer)' | sort
sudo dnf remove -y $(rpm -qa | grep -e '\.el[78]' | grep -vE '^(gpg-pubkey|libmodulemd|katello-ca-consumer)')

Retain the upgrade logs according to your organization's troubleshooting, audit, and retention requirements.

Review security settings and confirm that any temporary changes made during troubleshooting have been reversed where appropriate.

Restore normal monitoring and configuration-management operations.

Rollback

Leapp does not provide a built-in uninstall. Your rollback is whatever you prepared for earlier in this backup. Trust me I have seen situations where everything works fine but a single feature of the application which is not available after the upgrade was the reason for a rollback. Depending on what you planned for, you can:

• VM snapshot: revert the snapshot (accept the data loss since the snapshot was taken)
• LVM snapshot: merge the snapshot back and reboot
• ReaR or backup: restore the full system image Decide in advance what "point of no return" means: for example, "if the app is not healthy within 60 minutes of the reboot, revert." That decision is much easier before the maintenance window than during it.

Final Lessons and Recommendations

Congratulations you made it to this point. An in-place RHEL upgrade from RHEL 8 to 9 is a technical exercise, but its success depends just as much on preparation and coordination as it does on executing commands.

From planning to post-upgrade validation, administrators should focus on the following principles:

1. Plan before executing: Confirm the supported upgrade path, application compatibility, and maintenance window.
2. Back up and test recovery: Ensure that a reliable recovery option is available before making major system changes.
3. Use the pre-upgrade report: Resolve inhibitors and investigate warnings instead of treating the assessment as a formality.
4. Understand your dependencies: Third-party applications, drivers, repositories, and configuration-management tools can affect the outcome.
5. Maintain console access: Do not rely exclusively on SSH during an operation that changes core operating system components.
6. Validate business services: Confirm application functionality, networking, authentication, monitoring, and security after the upgrade.
7. Follow official documentation: Use the supported procedure for the exact source release, target release, architecture, and environment.

Conclusion

Upgrading from RHEL 8 to RHEL 9 is an important infrastructure lifecycle activity that can improve security, maintain supportability, and prepare systems for future technology requirements.

However, a successful upgrade requires more than running a few commands. It demands careful planning, compatibility checks, tested backups, systematic troubleshooting, and thorough post-upgrade validation.

Leapp provides a structured approach to supported in-place upgrades, but administrators must understand the environment and resolve the issues identified by the assessment before proceeding.

I am documenting this process to share practical lessons from Linux administration, help other engineers troubleshoot real-world challenges, and make enterprise operating system upgrades easier to plan and execute confidently.

The next in this series will be upgrading from RHEL 9 to 10.