DEV Community: Abhilash Kumar | Oracle ACE ♠

AUTOUPG_HOME - A smart way to manage Oracle GI & RU On Prem patches across versions for multiple database environments

Abhilash Kumar | Oracle ACE ♠ — Fri, 19 Jun 2026 14:21:47 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

Managing multiple Oracle patches

The purpose of this blog is to understand how patches are being managed even before patching.

More ofthen tha not you have identified the environment you want to patch , but when you have atleast 50+ databases to deal with do you have a way to manage patches , espeically when some databases still need to work on a slightly older releases due to application dependencies

It is difficult to remeber with hunderds of other things one has to remember , A typical problems in understanding patching nomenclature is below

p6880880_190000_Linux-ARM-64.zip --> This latest OPatch 

p38906621_190000_Linux-ARM-64.zip --> This is OJVM for 19.31

It's not possible to understand what the patch is for just by looking at the patch file name.

If you want to get started on autoupgrade , the below blog is a good read

Abhilash Kumar | Oracle ACE ♠ for Nabhaas Cloud Consulting

Jun 17

autougpgrade.jar - Provision & switch Oracle Homes ( Hands on lab scenario to move OH from 19.19 to 19.31 and rollback to 19.19 )

#oracle #oci #patch #rollback

31 min read

Reference : Official Doc on autoupgrade
https://docs.oracle.com/en/database/oracle/oracle-database/19/upgrd/about-oracle-database-autoupgrade.html

Enter AUTOUPG_HOME

AUTOUPG_HOME is a logical folder structure I jave created for my own efficiency to manage multiple patches and environments.

I have a base Autoupgrade Home /u01/autoupg-home with the following sub folders

autoupg-config - Contains all my config files for patching , upgrade , downloads and any config file I intend to use autoupgrade.jar for
autoupg-keystore - Contains the keystore for using autoupgrade.jar , typically you set it up once and forget about it.
autoupg-logs - Contains all log files generated by autoupgrade.jar
autoupg-sw - The repository for downloading patches , typically I would have sub folders under this for each GI & RU ( feel free to customize it as needed )

[oracle@oel8-lab01 ~]$ export AUTOUPG_HOME=/u01/autoupg-home
[oracle@oel8-lab01 ~]$ env | grep AUTO
AUTOUPG_HOME=/u01/autoupg-home

[oracle@oel8-lab01 ~]$ cd $AUTOUPG_HOME
drwxr-xr-x. 2 oracle oinstall      56 Jun 19 01:49 autoupg-config
drwxr-xr-x. 2 oracle oinstall      90 Jun 19 01:34 autoupg-keystore
drwxr-xr-x. 3 oracle oinstall      42 Jun 19 01:28 autoupg-logs
drwxr-xr-x. 6 oracle oinstall      70 Jun 19 01:24 autoupg-sw
-rw-r--r--. 1 oracle oinstall 7016621 Jun 19 01:20 autoupgrade.jar

A simple tree structure of AUTOUPG_HOME

[oracle@oel8-lab01 autoupg-home]$ tree
.
├── autoupg-config
│   ├── 19.31-RU-download.cfg
│   └── key-store.cfg
├── autoupg-keystore
│   ├── cwallet.sso
│   ├── cwallet.sso.lck
│   ├── ewallet.p12
│   └── ewallet.p12.lck
├── autoupg-logs
│   └── cfgtoollogs
│       └── patch
│           └── auto
│               ├── autoupgrade_patching_err.log
│               ├── autoupgrade_patching.log
│               ├── autoupgrade_patching_user.log
│               ├── config_files
│               │   └── autoupg_patching.cfg
│               ├── examiner
│               │   └── examiner2026_06_19_01_32_49.json
│               ├── lock
│               └── sql
│                   └── login.sql
├── autoupgrade.jar
├── autoupg-sw
│   ├── 19.28-RU
│   ├── 19.29-RU
│   ├── 19.30-RU
│   └── 19.31-RU
└── logs

Config Files & Download locations for Each GI/RU

I use in my lab Linux ARM64 so below is an example for downloading 19.91 RU for Linux ARM64

[oracle@oel8-lab01 autoupg-home]$ cat /u01/autoupg-home/autoupg-config/19.31-RU-download.cfg
global.global_log_dir=/u01/autoupg-home/autoupg-logs
global.keystore=/u01/autoupg-home/autoupg-keystore
install1.patch=OPATCH,OJVM,DPBP,AU,RU:19.31
install1.folder=/u01/autoupg-home/autoupg-sw/19.31-RU
install1.target_version=19
install1.platform=arm.x64

Downloading to desired folders

Now lets fire up the config file to get the dedicated patches downloaded to the desired locations.

[oracle@oel8-lab01 autoupg-home]$ java -jar ./autoupgrade.jar -config /u01/autoupg-home/autoupg-config/19.31-RU-download.cfg -patch -mode download
AutoUpgrade Patching 26.3.260401 launched with default internal options
Processing config file ...
Loading AutoUpgrade Patching keystore
AutoUpgrade Patching keystore is loaded

Connected to MOS - Searching for specified patches

----------------------------------------------------------
Downloading files to /u01/autoupg-home/autoupg-sw/19.31-RU
----------------------------------------------------------
DATABASE RELEASE UPDATE 19.31.0.0.0 (REL-APR2026)
    File: p39034528_190000_Linux-ARM-64.zip - VALIDATED

OPatch 12.2.0.1.51 for DB 19.0.0.0.0 (Apr 2026)
    File: p6880880_190000_Linux-ARM-64.zip - VALIDATED

OJVM RELEASE UPDATE 19.31.0.0.0
    File: p38906621_190000_Linux-ARM-64.zip - VALIDATED

DATAPUMP BUNDLE PATCH 19.31.0.0.0
    File: p39196236_1931000DBRU_Generic.zip - VALIDATED

autoupgrade.jar 26.3 (April 2026)
    File: autoupgrade.jar - VALIDATED
----------------------------------------------------------

Now Lets have a look at the AUTOUPG_HOME tree structure , you can see organized patches , this is pretty useful when dealing with large number of databases to have the patches available prior to downtime windows.

[oracle@oel8-lab01 autoupg-home]$ tree
.
├── autoupg-config
│   ├── 19.31-RU-download.cfg
│   └── key-store.cfg
├── autoupg-keystore
│   ├── apikey
│   ├── cwallet.sso
│   ├── cwallet.sso.lck
│   ├── ewallet.p12
│   └── ewallet.p12.lck
├── autoupg-logs
│   └── cfgtoollogs
│       └── patch
│           └── auto
│               ├── aru
│               │   ├── aru.log
│               │   ├── aru_releases.xml
│               │   ├── aru_user.log
│               │   └── ous.log
│               ├── autoupgrade_patching_err.log
│               ├── autoupgrade_patching.log
│               ├── autoupgrade_patching_user.log
│               ├── config_files
│               │   ├── autoupg_patching.cfg
│               │   └── autoupgrade.json
│               ├── examiner
│               │   ├── examiner2026_06_19_01_32_49.json
│               │   ├── examiner2026_06_19_01_57_59.json
│               │   ├── examiner2026_06_19_01_58_40.json
│               │   └── examiner2026_06_19_01_59_31.json
│               ├── lock
│               └── sql
│                   └── login.sql
├── autoupgrade.jar
├── autoupg-sw
│   ├── 19.28-RU
│   ├── 19.29-RU
│   ├── 19.30-RU
│   └── 19.31-RU
│       ├── autoupgrade.jar
│       ├── p38906621_190000_Linux-ARM-64.zip
│       ├── p39034528_190000_Linux-ARM-64.zip
│       ├── p39196236_1931000DBRU_Generic.zip
│       ├── p6880880_190000_Linux-ARM-64.zip
│       └── patches_info.json
└── logs

17 directories, 28 files
[oracle@oel8-lab01 autoupg-home]$

Patch Info

Note that each patch download has a file called patches_info.json, this a super useful feature to find description about the patches being downloaded

Use the below jq script

jq -r '
["TYPE","DESCRIPTION","PLATFORM","FILE","SHA1","SHA256"],
(.patches[] |
 [
   (if .description|test("RELEASE UPDATE") then "RU"
    elif .description|test("OPatch") then "OPATCH"
    elif .description|test("OJVM") then "OJVM"
    elif .description|test("DATAPUMP") then "DPBP"
    else "PATCH" end),
   .description,
   (.platform // "Generic"),
   .files[0].name,
   .files[0].checksum,
   .files[0]["checksum-256"]
 ]) | @tsv
' patches_info.json | column -t -s $'\t'

The tabular details of each patch that was downloaded in the corresponding folder is shown as below

TYPE    DESCRIPTION                                        PLATFORM          FILE                               SHA1                                      SHA256
RU      DATABASE RELEASE UPDATE 19.31.0.0.0 (REL-APR2026)  Linux ARM 64-bit  p39034528_190000_Linux-ARM-64.zip  35FCD029A7814A0960F1A0C700122D80CC1FA8EE  8B40287BA8B83746C527E285E039EE444F8E8E9A8391986EBF40588BBE46A92A
OPATCH  OPatch 12.2.0.1.51 for DB 19.0.0.0.0 (Apr 2026)    Linux ARM 64-bit  p6880880_190000_Linux-ARM-64.zip   4E6CB6C6F3FE33A092B2D570026596F4895E8605  D5D4692F89BE451C021F931125768BC68E5BEAF711FED7842361D65936EC19C2
RU      OJVM RELEASE UPDATE 19.31.0.0.0                    Linux ARM 64-bit  p38906621_190000_Linux-ARM-64.zip  A87F6A4B14D5851767CB5B1C7D089C1601E2F038  A4A8E7EFB5EB4BFE91C4A97C74A6702762A5AF2D8A5DEFEDB1042DD2F3E7BFB3
DPBP    DATAPUMP BUNDLE PATCH 19.31.0.0.0                  Generic           p39196236_1931000DBRU_Generic.zip  E711E3024AA6C4C336D7291CE1E7D0419C311C52  F39745219BCC9D8BC8FD1262EF787FF742F7F84F6C59A817B1F014ADA49B778F

So.. essentially organize all your patches before even you begin patching, that is what AUTOUPG_HOME helps to do.

oh_backup.sh - Oracle Home Backups for GI/RU

Abhilash Kumar | Oracle ACE ♠ — Wed, 17 Jun 2026 15:44:34 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

Oracle Home Backups - Why is it needed ?

As a part of pathcing Oracle DB / GI patches quarterly ( On Prem or OCI DB System or OCI EcaCS ) , a regular need to take a backup the Oracle Homes are needed as a best practice as mentioned in MOS Note : 565017.1
There needs to be a standard way these OH backups are taken so it is maintainable.
Repeatable use of OH backups using shell script capturing timestamp & IP of the nodes. If you are using multiple servers it becomes difficult to track which OH backup belongs to which server.

NOTE : Uncomment yne /u02 part of the script is you are using this for ExaCS

[root@oel8-lab01 ~]# nohup /u01/scripts/oh_backup.sh &

Script reference

A easy to use script called oh_backup.sh
https://github.com/abhilash-8/ora-tools/blob/master/oh_backup.sh

[root@oel8-lab01 ~]# cat /u01/scripts/oh_backup.sh 
#-------------------------------------------------------------------------------------------------
#--Script Name   : oh_backup.sh
#--Description   : Script to take Oracle Home Backup with timestamp ( On Prem & OCI )
#--Args          : None
#--Author        : Abhilash Kumar Bhattaram
#--Email         : abhilash8@gmail.com                                           
#-------------------------------------------------------------------------------------------------
# Run as Root user (Doc ID 565017.1)
# Replace /u01 and /u02 as applicable in your relevant environments.
export NOW=$(date +"%Y_%m_%d_%H%M%S")
export ip=`hostname -i`
export host=`hostname`
df -h /u01
ls -l /u01

### Tar Backup For DB & GI Home in /u01
export bkp_dir=/nfs_backup/oh_backup
cd /u01
tar -pcvf ${bkp_dir}/u01appbkp_${host}_${ip}_${NOW}.tar app 

### Specific to ExaCS
### Tar Backup For DB Home in /u02 
# Uncomment this part as applicable 
# cd /u02
# tar -pcvf ${bkp_dir}/u01appbkp_${host}_${ip}_${NOW}.tar app

autougpgrade.jar - Provision & switch Oracle Homes ( Hands on lab scenario to move OH from 19.19 to 19.31 and rollback to 19.19 )

Abhilash Kumar | Oracle ACE ♠ — Wed, 17 Jun 2026 09:37:39 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

Introduction about autoupgrade.jar

Autoupgrade is not only a tool for upgrading databases , it has a whole lot of functions such as

Upgrade databases ( automating several steps needed for an upgrade )
Provision Oracle Homes
Switch to a New Patched Oracle Home ( ( automating several steps needed for patching - Apply RU/OJVM/DPBP )
Download Oracle Patches
Provision New Oracle Homes
Unplug and Plug PDB's across CDB's
Convert Non CDB's to PDB's

So in short it is a highly versatile tool and the onus is on the DBA's to understand what use case works best the environment at hand

This the official documentation of autoupgrade.jar

https://docs.oracle.com/en/database/oracle/oracle-database/19/upgrd/about-oracle-database-autoupgrade.html

Scenario at hand - autoupgrade.jar provision new Oracle Homes and attempt a rollback to old Oracle Home

Lets consider the following scenario
I have a cdb1 which is on 19.19

[oracle@oel8-lab01 1_working]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 17 00:23:25 2026
Version 19.19.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1577055336 bytes
Fixed Size                  9163880 bytes
Variable Size             385875968 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7610368 bytes
Database mounted.
Database opened.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@oel8-lab01 1_working]$

[oracle@oel8-lab01 1_working]$ env | grep ORA
ORACLE_UNQNAME=cdb1
ORACLE_SID=cdb1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1
ORA_INVENTORY=/u01/app/oraInventory
ORACLE_HOSTNAME=oel8-lab01.nabhaas.com

/u01/app/oracle/product/19.0.0/dbhome_1 -- 19.19

I would like to do the following at a very high level

#1) Create 19.31 OH using autoupgrade.jar 
#2) Switch the 19.19 CDB to 19.31 OH
#3) Perform a downgrade back to 19.19

Pre-requisites

Before we start working on autoupgrade.jar , a few things to note about it.

Always use the latest autoupgrade.jar at all times to make use of the improved workflows in each version. The latest version is always available for download here - https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar

Below is the latest version of autoupgrade as on date.

[oracle@oel8-lab01 19c]$ java -jar autoupgrade.jar -version
build.version 26.3.260401
build.date 2026/04/01 20:09:34 +0000
build.hash b5bb11008
build.hash_date 2026/03/27 14:05:56 +0000
build.supported_target_versions 12.2,18,19,21,23,26
build.type production
build.label (HEAD, tag: v26.3, origin/stable_devel, stable_devel)
build.MOS_NOTE KB123450
build.MOS_LINK https://support.oracle.com/support/?anchorId=&kmContentId=2485457&page=sptemplate&sptemplate=km-article

autoupgrade.jar using openjdk8 at a minimum , I have used ( yum install openjdk ) to use for autoupgrade.jar , you could provision your own openjdk8 or JDK8 per your preference.
Always backup Oracle Homes before start of any activity
In general I use the following script as long as /u01 is dedicated for my DB and GI Homes , feel free modify the script as per your needs. This is based on MOS Note 565017.1

----- cat /home/oracle/scripts/oh_backup.sh 
# Run as Root user (Doc ID 565017.1)
export NOW=$(date +"%Y_%m_%d_%H%M%S")
export ip=`hostname -i`
export host=`hostname`
df -h /u01
ls -l /u01

### Tar Backup For DB & GI Home in /u01
export bkp_dir=/nfs_backup/oh_backup
cd /u01
tar -pcvf ${bkp_dir}/u01appbkp_${host}_${ip}_${NOW}.tar app

autoupgrade.jar - Provision new Oracle Homes

Since our DB Home for CDB is in 19.19 we need to have the following before proceeding to work on autoupgrade.jar. In this scenario download the patches and stage rather than autoupgrade.jar diwnloading it , would be the case in most regulated environments.

#1) A dedicated staging area for patches , in my case /u01/Disks/19c/

oracle@oel8-lab01 1_working]$ ls -l /u01/Disks/19c/
total 4127132
-rw-r--r--. 1 oracle oinstall    7016621 Jun 16 13:56 autoupgrade.jar
-rw-r--r--. 1 oracle oinstall        335 Jun 16 20:34 create-19-31.cfg
-rw-r--r--. 1 oracle oinstall 2415583176 Jun 16 11:38 LINUX.ARM64_1919000_db_home.zip
-rw-r--r--. 1 oracle oinstall        256 Jun 17 01:25 move-19-31.cfg
-rw-r--r--. 1 oracle oinstall        268 Jun 17 01:33 move-back-19-19.cfg
-rw-r--r--. 1 oracle oinstall 1734668614 Jun 16 11:38 p39034528_190000_Linux-ARM-64.zip
-rw-r--r--. 1 oracle oinstall   68892347 Jun 16 14:16 p6880880_190000_Linux-ARM-64.zip
[oracle@oel8-lab01 1_working]$ 


p39034528_190000_Linux-ARM-64.zip - My 19.31 patchset for Linux ARM-64
p6880880_190000_Linux-ARM-64.zip - My 19.31 patchset for Linux ARM-64

#2) Create Config files for Provisioning Oracle Home 19.31 - create-19-31.cfg

I have created an empty directory for /u01/app/oracle/product/19.0.0/dbhome_19-31 to stage my 19.31 OH 

[oracle@oel8-lab01 1_working]$ cat /u01/Disks/19c/create-19-31.cfg 
global.global_log_dir=/home/oracle/autoupgrade/logs
patch1.target_home=/u01/app/oracle/product/19.0.0/dbhome_19-31
patch1.folder=/u01/Disks/19c
patch1.patch=RU:19.31,OPATCH
patch1.home_settings.oracle_base=/u01/app/oracle
patch1.home_settings.edition=EE
patch1.home_settings.inventory_location=/u01/app/oraInventory
patch1.download=no

Lets analyze if my config file is fine using the -analyze

java -jar autoupgrade.jar -config create-19-31.cfg -patch -mode analyze

Now creating Oracle Homes using -mode create_home



[oracle@oel8-lab01 19c]$ java -jar autoupgrade.jar -config create-19-31.cfg -patch -mode create_home
AutoUpgrade Patching 26.3.260401 launched with default internal options
Processing config file ...

There were conditions found preventing AutoUpgrade Patching from successfully running

*Validating Oracle Patch files
A newer OPatch version file is not found in /u01/Disks/19c for the job with prefix patch1

[oracle@oel8-lab01 19c]$ java -jar autoupgrade.jar -config create-19-31.cfg -patch -mode create_home
Previous execution found loading latest data
Total jobs recovered: 1
+-----------------------------------------+
| Starting AutoUpgrade Patching execution |
+-----------------------------------------+
Type 'help' to list console commands
patch> ;sj
Unrecognized cmd: ;sj
patch> lsj
+----+-------------+-----------+---------+-------+----------+-------+----------------------------------------+
|Job#|      DB_NAME|      STAGE|OPERATION| STATUS|START_TIME|UPDATED|                                 MESSAGE|
+----+-------------+-----------+---------+-------+----------+-------+----------------------------------------+
| 100|create_home_1|OH_PATCHING|EXECUTING|RUNNING|  20:34:45| 5s ago|DATABASE RELEASE UPDATE 19.31.0.0.0 (REL|
+----+-------------+-----------+---------+-------+----------+-------+----------------------------------------+
Total jobs 1

patch> lsj
+----+-------------+-----------+---------+-------+----------+-------+----------------------------------------+
|Job#|      DB_NAME|      STAGE|OPERATION| STATUS|START_TIME|UPDATED|                                 MESSAGE|
+----+-------------+-----------+---------+-------+----------+-------+----------------------------------------+
| 100|create_home_1|OH_PATCHING|EXECUTING|RUNNING|  20:34:45|23s ago|DATABASE RELEASE UPDATE 19.31.0.0.0 (REL|
+----+-------------+-----------+---------+-------+----------+-------+----------------------------------------+


###### Checking Log files 

[oracle@oel8-lab01 bin]$ cat /home/oracle/autoupgrade/logs/cfgtoollogs/patch/auto/status/status.log
==========================================
   AutoUpgrade Patching Summary Report
==========================================
[Date]           Tue Jun 16 20:41:07 EDT 2026
[Number of Jobs] 1
==========================================
[Job ID] 100
==========================================
[DB Name]                create_home_1
[Version Before AutoUpgrade Patching] 19.0.0.0.0
[Version After AutoUpgrade Patching]  19.31.0.0.0
------------------------------------------
[Stage Name]    PENDING
[Status]        SUCCESS
[Start Time]    2026-06-16 20:34:46
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/pending
------------------------------------------
[Stage Name]    PREACTIONS
[Status]        SUCCESS
[Start Time]    2026-06-16 20:34:46
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/preaction
------------------------------------------
[Stage Name]    EXTRACT
[Status]        SUCCESS
[Start Time]    2026-06-16 20:34:46
[Duration]      0:00:29
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/extract
------------------------------------------
[Stage Name]    DBTOOLS
[Status]        SUCCESS
[Start Time]    2026-06-16 20:35:15
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/dbtools
------------------------------------------
[Stage Name]    INSTALL
[Status]        SUCCESS
[Start Time]    2026-06-16 20:35:16
[Duration]      0:00:33
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/install
------------------------------------------
[Stage Name]    OH_PATCHING
[Status]        SUCCESS
[Start Time]    2026-06-16 20:38:17
[Duration]      0:02:49
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/opatch
------------------------------------------
[Stage Name]    OPTIONS
[Status]        SUCCESS
[Start Time]    2026-06-16 20:41:07
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/options
------------------------------------------
[Stage Name]    ROOTSH
[Status]        SUCCESS
[Start Time]    2026-06-16 20:41:07
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/rootsh
------------------------------------------
[Stage Name]    DISPATCH
[Status]        SUCCESS
[Start Time]    2026-06-16 20:41:07
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/dispatch
------------------------------------------
[Stage Name]    POSTACTIONS
[Status]        SUCCESS
[Start Time]    2026-06-16 20:41:07
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/postaction
------------------------------------------
[oracle@oel8-lab01 bin]$

This shows the create new Oracle Homes 19.31 is created in /u01/app/oracle/product/19.0.0/dbhome_19-31

autoupgrade.jar - move to new Oracle Home provisioned

Now lets switch teh CDB with all its PDB's to the new Oracle home , its always a good idea to use analyze the config file

Below is the config file to move the CDB from 19.19 to 19.31

[oracle@oel8-lab01 19c]$ cat move-19-31.cfg
upg1.log_dir=/home/oracle/autoupgrade/logs
upg1.sid=cdb1
upg1.source_home=/u01/app/oracle/product/19.0.0/dbhome_1
upg1.target_home=/u01/app/oracle/product/19.0.0/dbhome_19-31
upg1.start_time=NOW
upg1.upgrade_node=oel8-lab01.nabhaas.com
upg1.target_version=19
[oracle@oel8-lab01 19c]$

java -jar autoupgrade.jar -config move-19-31.cfg -mode analyze

Now for our lab I induced a few failures to simulate troubleshooting with autoupgrade.jar

[oracle@oel8-lab01 19c]$ java -jar autoupgrade.jar -config move-19-31.cfg -mode analyze
No parameter 'global.global_log_dir' found in config file, using /u01/app/oracle/cfgtoollogs/autoupgrade
AutoUpgrade 26.3.260401 launched with default internal options
Processing config file ...
+--------------------------------+
| Starting AutoUpgrade execution |
+--------------------------------+
1 CDB(s) plus 2 PDB(s) will be analyzed
Type 'help' to list console commands
upg> 


[oracle@oel8-lab01 19c]$ java -jar autoupgrade.jar -config move-19-31.cfg -mode analyze
No parameter 'global.global_log_dir' found in config file, using /u01/app/oracle/cfgtoollogs/autoupgrade
AutoUpgrade 26.3.260401 launched with default internal options
Processing config file ...
+--------------------------------+
| Starting AutoUpgrade execution |
+--------------------------------+
1 CDB(s) plus 2 PDB(s) will be analyzed
Type 'help' to list console commands
upg> lsj
+----+-------+---------+---------+-------+----------+-------+----------------------------+
|Job#|DB_NAME|    STAGE|OPERATION| STATUS|START_TIME|UPDATED|                     MESSAGE|
+----+-------+---------+---------+-------+----------+-------+----------------------------+
| 100|   cdb1|PRECHECKS|EXECUTING|RUNNING|  00:04:44|12s ago|Loading database information|
+----+-------+---------+---------+-------+----------+-------+----------------------------+
Total jobs 1

upg> 

pg> logs
AutoUpgrade logs folder [/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto]
logs folder [cdb1][/home/oracle/autoupgrade/logs/cdb1]

upg> Job 100 completed
------------------- Final Summary --------------------
Number of databases            [ 1 ]

Jobs finished                  [1]
Jobs failed                    [0]

Please check the summary report at:
/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log
[oracle@oel8-lab01 19c]$ 


[oracle@oel8-lab01 19c]$ cat /u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log
==========================================
          Autoupgrade Summary Report
==========================================
[Date]           Wed Jun 17 00:05:25 EDT 2026
[Number of Jobs] 1
==========================================
[Job ID] 100
==========================================
[DB Name]                cdb1
[Version Before Upgrade] 19.19.0.0.0
[Version After Upgrade]  19.31.0.0.0
------------------------------------------
[Stage Name]    PRECHECKS
[Status]        FAILURE
[Start Time]    2026-06-17 00:04:44
[Duration]      0:00:40
[Log Directory] /home/oracle/autoupgrade/logs/cdb1/100/prechecks
[Detail]        /home/oracle/autoupgrade/logs/cdb1/100/prechecks/cdb1_preupgrade.log
                Check failed for CDB$ROOT, manual intervention needed for the below checks
                [FLASH_RECOVERY_AREA_SETUP, ARCHIVE_MODE_ON]
Cause:The following checks have ERROR severity and no auto fixup is available or
the fixup failed to resolve the issue. Fix them before continuing:
CDB$ROOT FLASH_RECOVERY_AREA_SETUP
CDB$ROOT ARCHIVE_MODE_ON
Reason:Database Checks has Failed details in /home/oracle/autoupgrade/logs/cdb1/100/prechecks
Action:[MANUAL]
Info:Return status is ERROR
ExecutionError:No
Error Message:The following checks have ERROR severity and no auto fixup is available or
the fixup failed to resolve the issue. Fix them before continuing:
CDB$ROOT FLASH_RECOVERY_AREA_SETUP
CDB$ROOT ARCHIVE_MODE_ON

------------------------------------------
[oracle@oel8-lab01 19c]$

As you can see the following are observed.

CDB in 19.19 must be archivelog mode ( as a Guaranteed Restore Point will be Created )
Flashback must be enabled in the database

So let's enable these in the CDB 19.19 OH where the database is currently residing in.



00:16:41 SQL> select instance_name, status, host_name, version, startup_time from gv$instance;

INSTANCE_NAME   |STATUS         |HOST_NAME                                                       |VERSION          |STARTUP_TIME
----------------|---------------|----------------------------------------------------------------|-----------------|--------------------
cdb1            |OPEN           |oel8-lab01.nabhaas.com                                          |19.0.0.0.0       |16-JUN-2026 23:56:52

00:16:41 SQL> select dbid, name, db_unique_name,flashback_on,force_logging,log_mode, open_mode, database_role, sysdate from gv$database;

      DBID|NAME     |DB_UNIQUE_NAME                |FLASHBACK_ON   |FORCE_LOGGING  |LOG_MODE       |OPEN_MODE           |DATABASE_ROLE       |SYSDATE
----------|---------|------------------------------|---------------|---------------|---------------|--------------------|--------------------|--------------------
1209803795|CDB1     |cdb1                          |NO             |NO             |NOARCHIVELOG   |READ WRITE          |PRIMARY             |17-JUN-2026 00:16:41


db1> show parameter reco

NAME                                |TYPE       |VALUE
------------------------------------|-----------|------------------------------
control_file_record_keep_time       |integer    |7
db_recovery_file_dest               |string     |
db_recovery_file_dest_size          |big integer|0
db_unrecoverable_scn_tracking       |boolean    |TRUE
recovery_parallelism                |integer    |0
remote_recovery_file_dest           |string     |


Elapsed: 00:00:00.00
cdb1> alter system set db_recovery_file_dest='/u01/oradata' scope=spfile;

System altered.

cdb1> alter system set db_recovery_file_dest_size=40g;

System altered.

Elapsed: 00:00:00.00
cdb1> show parameter reco

NAME                                |TYPE       |VALUE
------------------------------------|-----------|------------------------------
control_file_record_keep_time       |integer    |7
db_recovery_file_dest               |string     |
db_recovery_file_dest_size          |big integer|40G
db_unrecoverable_scn_tracking       |boolean    |TRUE
recovery_parallelism                |integer    |0
remote_recovery_file_dest           |string     |
cdb1> 


db1> shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
cdb1> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@oel8-lab01 1_working]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 17 00:23:25 2026
Version 19.19.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 1577055336 bytes
Fixed Size                  9163880 bytes
Variable Size             385875968 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7610368 bytes
Database mounted.
Database opened.
SQL> archive log list;
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     7
Current log sequence           9
SQL> shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@oel8-lab01 1_working]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 17 00:26:57 2026
Version 19.19.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1577055336 bytes
Fixed Size                  9163880 bytes
Variable Size             385875968 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7610368 bytes
Database mounted.
SQL> alter database archivelog;

Database altered.

SQL> alter database open;

Database altered.

SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     7
Next log sequence to archive   9
Current log sequence           9
SQL> show parameter reco

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time        integer     7
db_recovery_file_dest                string      /u01/oradata
db_recovery_file_dest_size           big integer 40G
db_unrecoverable_scn_tracking        boolean     TRUE
recovery_parallelism                 integer     0
remote_recovery_file_dest            string
SQL> 


cdb1> alter database flashback on;

Database altered.

Elapsed: 00:00:00.08
cdb1> @info

Session altered.

cdb1> alter database flashback on;

Database altered.

Elapsed: 00:00:00.08
cdb1> @info

Session altered.

Elapsed: 00:00:00.00
00:29:41 cdb1> select instance_name, status, host_name, version, startup_time from gv$instance;

INSTANCE_NAME   |STATUS         |HOST_NAME                                                       |VERSION          |STARTUP_TIME
----------------|---------------|----------------------------------------------------------------|-----------------|--------------------
cdb1            |OPEN           |oel8-lab01.nabhaas.com                                          |19.0.0.0.0       |17-JUN-2026 00:27:07

Elapsed: 00:00:00.00
00:29:41 cdb1> select dbid, name, db_unique_name,flashback_on,force_logging,log_mode, open_mode, database_role, sysdate from gv$database;

                DBID|NAME                                    |DB_UNIQUE_NAME                |FLASHBACK_ON   |FORCE_LOGGING  |LOG_MODE       |OPEN_MODE           |DATABASE_ROLE       |SYSDATE
--------------------|----------------------------------------|------------------------------|---------------|---------------|---------------|--------------------|--------------------|--------------------
          1209803795|CDB1                                    |cdb1                          |YES            |NO             |ARCHIVELOG     |READ WRITE          |PRIMARY             |17-JUN-2026 00:29:41



cdb1> -- Flash back Log Files
cdb1> set lines 1000
cdb1> COLUMN "Log NO" FOR 9,999
cdb1> COLUMN "Thread No" FOR 99
cdb1> COLUMN "Seq No" FOR 9999999999
cdb1> COLUMN "Size(GB)" FOR 999,999
cdb1> COLUMN "First Chg No" FOR 999,999,999,999,999,999
cdb1> col name for a80
cdb1>  SELECT
  2       log# as "Log No",
  3       thread# as "Thread No",
  4       sequence# as "Seq No",
  5       name,
  6       bytes/1024/1024/1024 as "Size(GB)",
  7       first_change# as "First Chg No",
  8       first_time
  9  FROM
 10     v$flashback_database_logfile
 11  ORDER BY first_time;

Log No|Thread No|     Seq No|NAME                                                                            |Size(GB)|            First Chg No|FIRST_TIME
------|---------|-----------|--------------------------------------------------------------------------------|--------|------------------------|--------------------
     1|        1|          1|/u01/oradata/CDB1/flashback/o1_mf_o348ohww_.flb                                 |       0|               1,243,005|17-JUN-2026 00:29:35
     2|        1|          1|/u01/oradata/CDB1/flashback/o1_mf_o348ojdj_.flb                                 |       0|                       0|

Elapsed: 00:00:00.01
cdb1> 
cdb1>

The above articulates archive logging flashback is enabled.

Lets retry the swicthing to new 19.31 which is /u01/app/oracle/product/19.0.0/dbhome_19-31

Retrying the same -analyze as earlier.

java -jar autoupgrade.jar -config move-19-31.cfg -mode analyze

We can now see the -analyze for moving to a 19.31 has not spotted any failures.


[oracle@oel8-lab01 19c]$ cat move-19-31.cfg 
upg1.log_dir=/home/oracle/autoupgrade/logs
upg1.sid=cdb1
upg1.source_home=/u01/app/oracle/product/19.0.0/dbhome_1
upg1.target_home=/u01/app/oracle/product/19.0.0/dbhome_19-31
upg1.start_time=NOW
upg1.upgrade_node=oel8-lab01.nabhaas.com
upg1.target_version=19
[oracle@oel8-lab01 19c]$ java -jar autoupgrade.jar -config move-19-31.cfg -mode analyze
No parameter 'global.global_log_dir' found in config file, using /u01/app/oracle/cfgtoollogs/autoupgrade
AutoUpgrade 26.3.260401 launched with default internal options
Processing config file ...
+--------------------------------+
| Starting AutoUpgrade execution |
+--------------------------------+
1 CDB(s) plus 2 PDB(s) will be analyzed
Type 'help' to list console commands
upg> lsj
+----+-------+---------+---------+-------+----------+-------+----------------------------+
|Job#|DB_NAME|    STAGE|OPERATION| STATUS|START_TIME|UPDATED|                     MESSAGE|
+----+-------+---------+---------+-------+----------+-------+----------------------------+
| 101|   cdb1|PRECHECKS|EXECUTING|RUNNING|  00:34:51| 1s ago|Loading database information|
+----+-------+---------+---------+-------+----------+-------+----------------------------+
Total jobs 1

upg> logs
AutoUpgrade logs folder [/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto]
logs folder [cdb1][/home/oracle/autoupgrade/logs/cdb1]


upg> Job 101 completed
------------------- Final Summary --------------------
Number of databases            [ 1 ]

Jobs finished                  [1]
Jobs failed                    [0]

Please check the summary report at:
/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log
[oracle@oel8-lab01 19c]$ 

upg> 


oracle@oel8-lab01 auto]$ cat /u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log
==========================================
          Autoupgrade Summary Report
==========================================
[Date]           Wed Jun 17 00:35:39 EDT 2026
[Number of Jobs] 1
==========================================
[Job ID] 101
==========================================
[DB Name]                cdb1
[Version Before Upgrade] 19.19.0.0.0
[Version After Upgrade]  19.31.0.0.0
------------------------------------------
[Stage Name]    PRECHECKS
[Status]        SUCCESS
[Start Time]    2026-06-17 00:34:51
[Duration]      0:00:47
[Log Directory] /home/oracle/autoupgrade/logs/cdb1/101/prechecks
[Detail]        /home/oracle/autoupgrade/logs/cdb1/101/prechecks/cdb1_preupgrade.log
                Check passed and no manual intervention needed
------------------------------------------
[oracle@oel8-lab01 auto]$

Now lets move the CDB from 19.19 to 19.31 using the -deploy option

java -jar autoupgrade.jar -config move-19-31.cfg -mode deploy

### No changes in config file , showing here for reference
[oracle@oel8-lab01 19c]$ cat move-19-31.cfg 
upg1.log_dir=/home/oracle/autoupgrade/logs
upg1.sid=cdb1
upg1.source_home=/u01/app/oracle/product/19.0.0/dbhome_1
upg1.target_home=/u01/app/oracle/product/19.0.0/dbhome_19-31
upg1.start_time=NOW
upg1.upgrade_node=oel8-lab01.nabhaas.com
upg1.target_version=19

Running the -deploy
Some commands for reference as soon as the autoupgrade.jar kicks off

lsj [ List autoupgrade Jobs ]
lsj -a [ List autoupgrade jobs repeatedly to monitor the jobs ]

[oracle@oel8-lab01 19c]$ java -jar autoupgrade.jar -config move-19-31.cfg -mode deploy
No parameter 'global.global_log_dir' found in config file, using /u01/app/oracle/cfgtoollogs/autoupgrade
AutoUpgrade 26.3.260401 launched with default internal options
Processing config file ...
+--------------------------------+
| Starting AutoUpgrade execution |
+--------------------------------+
1 CDB(s) plus 2 PDB(s) will be processed
Type 'help' to list console commands
upg> lsj
+----+-------+---------+---------+-------+----------+-------+----------------------------+
|Job#|DB_NAME|    STAGE|OPERATION| STATUS|START_TIME|UPDATED|                     MESSAGE|
+----+-------+---------+---------+-------+----------+-------+----------------------------+
| 102|   cdb1|PRECHECKS|EXECUTING|RUNNING|  00:40:01| 2s ago|Loading database information|
+----+-------+---------+---------+-------+----------+-------+----------------------------+
Total jobs 1

upg> logs
AutoUpgrade logs folder [/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto]
logs folder [cdb1][/home/oracle/autoupgrade/logs/cdb1]

upg> lsj
+----+-------+---------+---------+-------+----------+-------+----------------+
|Job#|DB_NAME|    STAGE|OPERATION| STATUS|START_TIME|UPDATED|         MESSAGE|
+----+-------+---------+---------+-------+----------+-------+----------------+
| 102|   cdb1|PREFIXUPS|EXECUTING|RUNNING|  00:40:01| 2s ago|Executing fixups|
+----+-------+---------+---------+-------+----------+-------+----------------+
Total jobs 1

upg> lsj
+----+-------+---------+---------+-------+----------+-------+----------------+
|Job#|DB_NAME|    STAGE|OPERATION| STATUS|START_TIME|UPDATED|         MESSAGE|
+----+-------+---------+---------+-------+----------+-------+----------------+
| 102|   cdb1|PREFIXUPS|EXECUTING|RUNNING|  00:40:01|10s ago|Executing fixups|
+----+-------+---------+---------+-------+----------+-------+----------------+
Total jobs 1

upg> 

pg> lsj -a
upg> +----+-------+---------+---------+-------+----------+-------+--------------------+
|Job#|DB_NAME|    STAGE|OPERATION| STATUS|START_TIME|UPDATED|             MESSAGE|
+----+-------+---------+---------+-------+----------+-------+--------------------+
| 102|   cdb1|DBUPGRADE|EXECUTING|RUNNING|  00:40:01|41s ago|12%Patching CDB$ROOT|
+----+-------+---------+---------+-------+----------+-------+--------------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+---------+---------+-------+----------+-------+--------------------+
|Job#|DB_NAME|    STAGE|OPERATION| STATUS|START_TIME|UPDATED|             MESSAGE|
+----+-------+---------+---------+-------+----------+-------+--------------------+
| 102|   cdb1|DBUPGRADE|EXECUTING|RUNNING|  00:40:01|48s ago|12%Patching CDB$ROOT|
+----+-------+---------+---------+-------+----------+-------+--------------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT


The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+----------+---------+-------+----------+-------+----------------+
|Job#|DB_NAME|     STAGE|OPERATION| STATUS|START_TIME|UPDATED|         MESSAGE|
+----+-------+----------+---------+-------+----------+-------+----------------+
| 102|   cdb1|POSTFIXUPS|EXECUTING|RUNNING|  00:40:01| 6s ago|Executing fixups|
+----+-------+----------+---------+-------+----------+-------+----------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+----------+---------+-------+----------+-------+----------------+
|Job#|DB_NAME|     STAGE|OPERATION| STATUS|START_TIME|UPDATED|         MESSAGE|
+----+-------+----------+---------+-------+----------+-------+----------------+
| 102|   cdb1|POSTFIXUPS|EXECUTING|RUNNING|  00:40:01| 1s ago|Executing fixups|
+----+-------+----------+---------+-------+----------+-------+----------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+----------+---------+-------+----------+-------+----------------+
|Job#|DB_NAME|     STAGE|OPERATION| STATUS|START_TIME|UPDATED|         MESSAGE|
+----+-------+----------+---------+-------+----------+-------+----------------+
| 102|   cdb1|POSTFIXUPS|EXECUTING|RUNNING|  00:40:01| 0s ago|Executing fixups|
+----+-------+----------+---------+-------+----------+-------+----------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+----------+---------+-------+----------+-------+----------------+
|Job#|DB_NAME|     STAGE|OPERATION| STATUS|START_TIME|UPDATED|         MESSAGE|
+----+-------+----------+---------+-------+----------+-------+----------------+
| 102|   cdb1|POSTFIXUPS|EXECUTING|RUNNING|  00:40:01| 7s ago|Executing fixups|
+----+-------+----------+---------+-------+----------+-------+----------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+----------+---------+-------+----------+-------+----------------+
|Job#|DB_NAME|     STAGE|OPERATION| STATUS|START_TIME|UPDATED|         MESSAGE|
+----+-------+----------+---------+-------+----------+-------+----------------+
| 102|   cdb1|POSTFIXUPS|EXECUTING|RUNNING|  00:40:01| 5s ago|Executing fixups|
+----+-------+----------+---------+-------+----------+-------+----------------+
Total jobs 1
.
.
.
.
.
.

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+----------+---------+-------+----------+-------+------------------+
|Job#|DB_NAME|     STAGE|OPERATION| STATUS|START_TIME|UPDATED|           MESSAGE|
+----+-------+----------+---------+-------+----------+-------+------------------+
| 102|   cdb1|POSTFIXUPS|EXECUTING|RUNNING|  00:40:01| 2s ago|Refreshing DB info|
+----+-------+----------+---------+-------+----------+-------+------------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+----------+---------+-------+----------+-------+------------------+
|Job#|DB_NAME|     STAGE|OPERATION| STATUS|START_TIME|UPDATED|           MESSAGE|
+----+-------+----------+---------+-------+----------+-------+------------------+
| 102|   cdb1|POSTFIXUPS|EXECUTING|RUNNING|  00:40:01| 9s ago|Refreshing DB info|
+----+-------+----------+---------+-------+----------+-------+------------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+----------+---------+-------+----------+-------+-------------------+
|Job#|DB_NAME|     STAGE|OPERATION| STATUS|START_TIME|UPDATED|            MESSAGE|
+----+-------+----------+---------+-------+----------+-------+-------------------+
| 102|   cdb1|POSTFIXUPS|EXECUTING|RUNNING|  00:40:01| 0s ago|Re-Executing Checks|
+----+-------+----------+---------+-------+----------+-------+-------------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
+----+-------+----------+---------+-------+----------+-------+-------------------+
|Job#|DB_NAME|     STAGE|OPERATION| STATUS|START_TIME|UPDATED|            MESSAGE|
+----+-------+----------+---------+-------+----------+-------+-------------------+
| 102|   cdb1|POSTFIXUPS|EXECUTING|RUNNING|  00:40:01| 1s ago|Re-Executing Checks|
+----+-------+----------+---------+-------+----------+-------+-------------------+
Total jobs 1

The command lsj is running every 7 seconds. PRESS ENTER TO EXIT
Job 102 completed
------------------- Final Summary --------------------
Number of databases            [ 1 ]

Jobs finished                  [1]
Jobs failed                    [0]
Jobs restored                  [0]
Jobs pending                   [0]

---- Drop GRP at your convenience once you consider it is no longer needed ----
Drop GRP from cdb1: drop restore point AUTOUPGRADE_9212_CDB11919000


Please check the summary report at:
/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.html
/u01/app/oracle/cfgtoollogs/autoupgrade/cfgtoollogs/upgrade/auto/status/status.log
[oracle@oel8-lab01 19c]$

Checking autoupgrade log files , this shows switching to 19.30 has gone well



[oracle@oel8-lab01 102]$ pwd
/home/oracle/autoupgrade/logs/cdb1/102
[oracle@oel8-lab01 102]$ tail -50f autoupgrade_20260617.log
.
.


2026-06-17 00:54:56.802 INFO [81] Completed retrieving next JobStage - JobDefinition.getNextStage#101 
2026-06-17 00:54:56.802 INFO [81] Starting background threads for [oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteSysUpdates] - JobStage.startThreads#160 
2026-06-17 00:54:56.802 INFO [81] Completed starting background threads for [oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteSysUpdates] - JobStage.startThreads#162 
2026-06-17 00:54:56.806 INFO [81] Executing stage SYSUPDATES of job 102 - RunJobDefinition.runJob#141 
2026-06-17 00:54:56.806 INFO [81] Running JobStageAction [oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteSysUpdates] - JobStage.executeStage#78 
2026-06-17 00:54:57.672 INFO [81] The status of the JobStageAction [oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteSysUpdates] was [true] - JobStage.executeStage#86 
2026-06-17 00:54:57.675 INFO [81] Completed stage SYSUPDATES of job 102 - RunJobDefinition.runJob#149 
2026-06-17 00:54:57.675 INFO [81] Stopping background threads for [oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteSysUpdates] - JobStage.stopThreads#169 
2026-06-17 00:54:57.675 INFO [81] Completed stopping background threads for [oracle.upgrade.autoupgrade.dispatcher.facade.subsystems.ExecuteSysUpdates] - JobStage.stopThreads#171 
2026-06-17 00:54:57.677 INFO [81] Adding Job context key[dispatcher_stage] value[13] for job [102] - CommonBackBone.addJobContext#270 
2026-06-17 00:54:57.679 INFO [81] Stopping background threads for Job - JobDefinition.stopJobThreads#132 
2026-06-17 00:54:57.679 INFO [81] [0] threads have been stopped by JobDefinition - JobDefinition.stopJobThreads#134 
2026-06-17 00:54:57.679 INFO [81] Dispatcher finished for 102 - RunJobDefinition.runJob#177 
2026-06-17 00:54:57.679 INFO [81] End  - RunJobDefinition.runJob#178 
2026-06-17 00:54:57.688 INFO [81] ----------------------Stages  Summary------------------------ - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     SETUP             <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     GRP               <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     PREUPGRADE        <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     PRECHECKS         1 min                                   - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     PREFIXUPS         2 min                                   - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     DRAIN             <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     DBUPGRADE         9 min                                   - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     DISPATCH          <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     POSTCHECKS        <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     DISPATCH          <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     POSTFIXUPS        1 min                                   - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     POSTUPGRADE       <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     SYSUPDATES        <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81]     COMPLETED         <1 min                                  - Utilities.writeStageSummary#1157 
2026-06-17 00:54:57.688 INFO [81] End of dispatcher instance for cdb1 - CommonBackBone.finalJobLogging#436 
^C
[oracle@oel8-lab01 102]$ pwd

Checking Oracle DB versions and dba_registry , we observe the following

SQL*Plus Version has changed to 19.31
dba_registry has all components showing to 19.31
Guaranteed restore point AUTOUPGRADE_9212_CDB11919000 was created prior to starting the switching Oracle Home
We can see flashback logs were created for the above Guaranteed restore point


[oracle@oel8-lab01 19c]$ echo $ORACLE_HOME
/u01/app/oracle/product/19.0.0/dbhome_1
[oracle@oel8-lab01 19c]$ export ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_19-31
[oracle@oel8-lab01 1_working]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 17 01:00:06 2026
Version 19.19.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.31.0.0.0

SQL> @info

Session altered.

01:00:11 SQL> select instance_name, status, host_name, version, startup_time from gv$instance;

INSTANCE_NAME   |STATUS         |HOST_NAME                                                       |VERSION          |STARTUP_TIME
----------------|---------------|----------------------------------------------------------------|-----------------|--------------------
cdb1            |OPEN           |oel8-lab01.nabhaas.com                                          |19.0.0.0.0       |17-JUN-2026 00:44:14

01:00:11 SQL> select dbid, name, db_unique_name,flashback_on,force_logging,log_mode, open_mode, database_role, sysdate from gv$database;

      DBID|NAME     |DB_UNIQUE_NAME                |FLASHBACK_ON   |FORCE_LOGGING  |LOG_MODE       |OPEN_MODE           |DATABASE_ROLE       |SYSDATE
----------|---------|------------------------------|---------------|---------------|---------------|--------------------|--------------------|--------------------
1209803795|CDB1     |cdb1                          |YES            |YES            |ARCHIVELOG     |READ WRITE          |PRIMARY             |17-JUN-2026 01:00:11



SQL> select * from dba_registry;

COMP_ID
------------------------------
COMP_NAME
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
VERSION                       |VERSION_FULL                  |STATUS         |MODIFIED                     |NAMESPACE
------------------------------|------------------------------|---------------|-----------------------------|------------------------------
CONTROL
--------------------------------------------------------------------------------------------------------------------------------
SCHEMA
--------------------------------------------------------------------------------------------------------------------------------
PROCEDURE                                                                                                                       |STARTUP |PARENT_ID
--------------------------------------------------------------------------------------------------------------------------------|--------|------------------------------
OTHER_SCHEMAS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CATALOG
Oracle Database Catalog Views
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:30         |SERVER
SYS
SYS
DBMS_REGISTRY_SYS.VALIDATE_CATALOG                                                                                              |        |


CATPROC
Oracle Database Packages and Types
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:30         |SERVER
SYS
SYS
DBMS_REGISTRY_SYS.VALIDATE_CATPROC                                                                                              |        |
APPQOSSYS,AUDSYS,DBSFWUSER,DBSNMP,DIP,GGSYS,GSMADMIN_INTERNAL,GSMCATUSER,GSMROOTUSER,GSMUSER,ORACLE_OCM,OUTLN,REMOTE_SCHEDULER_AGENT,SYS$UMF,SYSBACKUP,SYSDG,SYSKM,SYSRAC,SYSTEM

RAC
Oracle Real Application Clusters
19.0.0.0.0                    |19.31.0.0.0                   |OPTION OFF     |09-JUN-2023 00:12:23         |SERVER
SYS
SYS
DBMS_CLUSTDB.VALIDATE                                                                                                           |        |


JAVAVM
JServer JAVA Virtual Machine
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
SYS
INITJVMAUX.VALIDATE_JAVAVM                                                                                                      |        |
OJVMSYS

XML
Oracle XDK
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
SYS
XMLVALIDATE                                                                                                                     |        |


CATJAVA
Oracle Database Java Packages
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
SYS
DBMS_REGISTRY_SYS.VALIDATE_CATJAVA                                                                                              |        |


APS
OLAP Analytic Workspace
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
SYS
APS_VALIDATE                                                                                                                    |        |


XDB
Oracle XML Database
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:30         |SERVER
SYS
XDB
DBMS_REGXDB.VALIDATEXDB                                                                                                         |        |
ANONYMOUS,XS$NULL

OWM
Oracle Workspace Manager
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
WMSYS
VALIDATE_OWM                                                                                                                    |        |


CONTEXT
Oracle Text
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
CTXSYS
VALIDATE_CONTEXT                                                                                                                |        |


ORDIM
Oracle Multimedia
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
ORDSYS
VALIDATE_ORDIM                                                                                                                  |        |
MDSYS,ORDDATA,ORDPLUGINS,SI_INFORMTN_SCHEMA

SDO
Spatial
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
MDSYS
VALIDATE_SDO                                                                                                                    |        |
MDDATA,MDSYS

DV
Oracle Database Vault
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
DVSYS
VALIDATE_DV                                                                                                                     |        |
DVF

XOQ
Oracle OLAP API
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
OLAPSYS
XOQ_VALIDATE                                                                                                                    |        |


OLS
Oracle Label Security
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
LBACSYS
VALIDATE_OLS                                                                                                                    |        |


15 rows selected.


01:19:01 SQL> select * from v$flash_Recovery_area_usage;

FILE_TYPE           |PERCENT_SPACE_USED|PERCENT_SPACE_RECLAIMABLE|NUMBER_OF_FILES|    CON_ID
--------------------|------------------|-------------------------|---------------|----------
CONTROL FILE        |                 0|                        0|              0|         0
REDO LOG            |                 0|                        0|              0|         0
ARCHIVED LOG        |              3.51|                        0|             33|         0
BACKUP PIECE        |               .04|                        0|              1|         0
IMAGE COPY          |                 0|                        0|              0|         0
FLASHBACK LOG       |              1.46|                        0|             12|         0   <<<<<----
FOREIGN ARCHIVED LOG|                 0|                        0|              0|         0
AUXILIARY DATAFILE C|                 0|                        0|              0|         0
OPY                 |                  |                         |               |



Elapsed: 00:00:00.02
cdb1> SELECT NAME, SCN, TIME, GUARANTEE_FLASHBACK_DATABASE FROM V$RESTORE_POINT;

NAME                                    |                 SCN|TIME                                                                       |GUA
----------------------------------------|--------------------|---------------------------------------------------------------------------|---
AUTOUPGRADE_9212_CDB11919000            |             1244196|17-JUN-26 12.40.02.000000000 AM                                            |YES


cdb1> drop restore point AUTOUPGRADE_9212_CDB11919000;

Restore point dropped.

We have ensured new DB Home has moved to /u01/app/oracle/product/19.0.0/dbhome_19-31 from /u01/app/oracle/product/19.0.0/dbhome_1

Now it's all rainbows and roses until you want to perform a rollback.. which could be due to various reasons...

Appling RU patches have a history of inducing internal ORA-00600 errors and amounting to db restarts leading to unwarranted downtime
RU Patches have to go through rigirous tetsing in Non Prod Applications in Non production . there might me some integrated application that may not be supporting the latest release.

Though DBA's would have applied patches without issues in Non Production , there are always cases where applying patches have been counter productive , but thats the challenge we need to move ahead of these patches continue applying latest RU's considering the larger risks of not applying patches.

Rollback - Precautions to be taken before any rollback

Whenever you have a rollback situation ( rolling back to older DB RU's ), ensure you have the following

Raise an MOS SR as a Priority 1 , there might be one off patches which Oracle has already identified with various customers on what you are facing.
If Rollback is mandatory , provision the working Orale Home taken prior to switching to new Oracle Homes , ensure your environment variables are pointing to the 19.19 OH.

Rollback - Use Oracle Documented methods

Every Database release has documented steps on how to downgrade to older Oracle releases

https://docs.oracle.com/en/database/oracle/oracle-database/19/upgrd/downgrading-oracle-db-after-upgrade.html#GUID-EAF290FD-5F22-4FEE-829D-B3587168471F

To your curiosity does autoupgrade allow moving to older Oracle Homes , I checked it does not allow.

[oracle@oel8-lab01 19c]$ cat move-back-19-19.cfg
global.global_log_dir=/home/oracle/autoupgrade/logs
upg1.sid=cdb1
upg1.source_home=/u01/app/oracle/product/19.0.0/dbhome_19-31
upg1.target_home=/u01/app/oracle/product/19.0.0/dbhome_1
upg1.start_time=NOW
upg1.upgrade_node=oel8-lab01.nabhaas.com
upg1.target_version=19

[oracle@oel8-lab01 19c]$ java -jar autoupgrade.jar -config move-back-19-19.cfg -mode analyze
AutoUpgrade 26.3.260401 launched with default internal options
Processing config file ...
upg1
Source Home RU 19.31.0.0.0 version is greater than Target Home RU 19.19.0.0.0 version
RU apply configuration is invalid for entry upg1

[oracle@oel8-lab01 19c]$

The below are the documented steps I had used to move back from 19.31 to 19.19

Check if datapatch has been run before rolling back.


[oracle@oel8-lab01 19c]$ $ORACLE_HOME/OPatch/datapatch -verbose
SQL Patching tool version 19.31.0.0.0 Production on Wed Jun 17 01:35:45 2026
Copyright (c) 2012, 2026, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_15854_2026_06_17_01_35_45/sqlpatch_invocation.log

Connecting to database...OK
Gathering database info...done

Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)

Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of interim SQL patches:
  No interim patches found

Current state of release update SQL patches:
  Binary registry:
    19.31.0.0.0 Release_Update 260515124233: Installed
  PDB CDB$ROOT:
    Applied 19.31.0.0.0 Release_Update 260515124233 successfully on 17-JUN-26 12.47.08.494152 AM
  PDB PDB$SEED:
    Applied 19.31.0.0.0 Release_Update 260515124233 successfully on 17-JUN-26 12.53.30.055176 AM
  PDB PDB1:
    Applied 19.31.0.0.0 Release_Update 260515124233 successfully on 17-JUN-26 12.53.30.069902 AM

Adding patches to installation queue and performing prereq checks...done
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED PDB1
    No interim patches need to be rolled back
    No release update patches need to be installed
    No interim patches need to be applied

SQL Patching tool complete on Wed Jun 17 01:36:01 2026
[oracle@oel8-lab01 19c]$

Before downgrading the database to older OH (19.19 in our case) Check the dba_registry after running the datapatch , after restarting the database.

Check for Database Vaults if any and clear the dab_audit_trail.


cdb1> SELECT * FROM DBA_DV_STATUS;

NAME                                    |STATUS
----------------------------------------|---------------
DV_APP_PROTECTION                       |NOT CONFIGURED
DV_CONFIGURE_STATUS                     |FALSE
DV_ENABLE_STATUS                        |FALSE

Elapsed: 00:00:00.01
cdb1> SELECT * FROM CDB_DV_STATUS;

NAME                                    |STATUS         |              CON_ID
----------------------------------------|---------------|--------------------
DV_APP_PROTECTION                       |NOT CONFIGURED |                   1
DV_CONFIGURE_STATUS                     |FALSE          |                   1
DV_ENABLE_STATUS                        |FALSE          |                   1
DV_APP_PROTECTION                       |NOT CONFIGURED |                   3
DV_CONFIGURE_STATUS                     |FALSE          |                   3
DV_ENABLE_STATUS                        |FALSE          |                   3

6 rows selected.

Elapsed: 00:00:00.01
cdb1> SELECT COUNT(*) FROM UNIFIED_AUDIT_TRAIL;

            COUNT(*)
--------------------
                4240

Elapsed: 00:00:00.03
cdb1>  CREATE TABLE UA_DATA AS (SELECT * FROM UNIFIED_AUDIT_TRAIL);

Table created.

Elapsed: 00:00:00.76
cdb1> EXEC DBMS_AUDIT_MGMT.CLEAN_AUDIT_TRAIL(audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_UNIFIED, use_last_arch_timestamp => FALSE);

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.12
cdb1> select * from  V$TIMEZONE_FILE;

FILENAME            |             VERSION|              CON_ID
--------------------|--------------------|--------------------
timezlrg_32.dat     |                  32|                   0

Elapsed: 00:00:00.00
cdb1> show parameter spfile

NAME                                |TYPE       |VALUE
------------------------------------|-----------|------------------------------
spfile                              |string     |/u01/app/oracle/product/19.0.0
                                    |           |/dbhome_1/dbs/spfilecdb1.ora

Create a pfile from spfile for the purpose of downgrade and start the CDB's and PDB's in the 19.19 OH.

NOTE : You will still see 19.31 for all components in dba_registry , this needs to be fixed toi 19.19

cdb1> create pfile='/home/oracle/downgrade.ora' from spfile;

File created.

Elapsed: 00:00:00.00
cdb1> shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
cdb1> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0
[oracle@oel8-lab01 1_working]$ 


SQL> startup downgrade pfile='/home/oracle/downgrade.ora'
SQL> alter pluggable database all open downgrade;


oracle@oel8-lab01 1_working]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jun 17 02:18:46 2026
Version 19.19.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup downgrade pfile='/home/oracle/downgrade.ora';
ORACLE instance started.

Total System Global Area 1577055336 bytes
Fixed Size                  9163880 bytes
Variable Size             402653184 bytes
Database Buffers         1157627904 bytes
Redo Buffers                7610368 bytes
Database mounted.
Database opened.
SQL> alter pluggable database all open downgrade;

Pluggable database altered.

SQL> 

cdb1> select * from dba_registry;

COMP_ID
------------------------------
COMP_NAME
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
VERSION                       |VERSION_FULL                  |STATUS         |MODIFIED                     |NAMESPACE
------------------------------|------------------------------|---------------|-----------------------------|------------------------------
CONTROL
--------------------------------------------------------------------------------------------------------------------------------
SCHEMA
--------------------------------------------------------------------------------------------------------------------------------
PROCEDURE                                                                                                                       |STARTUP |PARENT_ID
--------------------------------------------------------------------------------------------------------------------------------|--------|------------------------------
OTHER_SCHEMAS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CATALOG
Oracle Database Catalog Views
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:30         |SERVER
SYS
SYS
DBMS_REGISTRY_SYS.VALIDATE_CATALOG                                                                                              |        |


CATPROC
Oracle Database Packages and Types
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:30         |SERVER
SYS
SYS
DBMS_REGISTRY_SYS.VALIDATE_CATPROC                                                                                              |        |
APPQOSSYS,AUDSYS,DBSFWUSER,DBSNMP,DIP,GGSYS,GSMADMIN_INTERNAL,GSMCATUSER,GSMROOTUSER,GSMUSER,ORACLE_OCM,OUTLN,REMOTE_SCHEDULER_AGENT,SYS$UMF,SYSBACKUP,SYSDG,SYSKM,SYSRAC,SYSTEM

RAC
Oracle Real Application Clusters
19.0.0.0.0                    |19.31.0.0.0                   |OPTION OFF     |09-JUN-2023 00:12:23         |SERVER
SYS
SYS
DBMS_CLUSTDB.VALIDATE                                                                                                           |        |


JAVAVM
JServer JAVA Virtual Machine
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
SYS
INITJVMAUX.VALIDATE_JAVAVM                                                                                                      |        |
OJVMSYS

XML
Oracle XDK
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
SYS
XMLVALIDATE                                                                                                                     |        |


CATJAVA
Oracle Database Java Packages
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
SYS
DBMS_REGISTRY_SYS.VALIDATE_CATJAVA                                                                                              |        |


APS
OLAP Analytic Workspace
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
SYS
APS_VALIDATE                                                                                                                    |        |


XDB
Oracle XML Database
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:30         |SERVER
SYS
XDB
DBMS_REGXDB.VALIDATEXDB                                                                                                         |        |
ANONYMOUS,XS$NULL

OWM
Oracle Workspace Manager
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
WMSYS
VALIDATE_OWM                                                                                                                    |        |


CONTEXT
Oracle Text
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
CTXSYS
VALIDATE_CONTEXT                                                                                                                |        |


ORDIM
Oracle Multimedia
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
ORDSYS
VALIDATE_ORDIM                                                                                                                  |        |
MDSYS,ORDDATA,ORDPLUGINS,SI_INFORMTN_SCHEMA

SDO
Spatial
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
MDSYS
VALIDATE_SDO                                                                                                                    |        |
MDDATA,MDSYS

DV
Oracle Database Vault
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
DVSYS
VALIDATE_DV                                                                                                                     |        |
DVF

XOQ
Oracle OLAP API
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
OLAPSYS
XOQ_VALIDATE                                                                                                                    |        |


OLS
Oracle Label Security
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
LBACSYS
VALIDATE_OLS                                                                                                                    |        |



15 rows selected.

Use the dbdowngrade , which by default will not have execute permissions




[oracle@oel8-lab01 1_working]$ cd $ORACLE_HOME/bin
[oracle@oel8-lab01 bin]$ ls -l dbdowngrade
-rw-r--r--. 1 oracle oinstall 8336 Feb  9  2018 dbdowngrade
[oracle@oel8-lab01 bin]$ ./dbdowngrade
-bash: ./dbdowngrade: Permission denied
[oracle@oel8-lab01 bin]$ chmod +x dbdowngrade
[oracle@oel8-lab01 bin]$ ./dbdowngrade
Downgrading containers
catcon::set_log_file_base_path: ALL catcon-related output will be written to [/u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/downgrade/catdwgrd_catcon_17230.lst]

catcon::set_log_file_base_path: catcon: See [/u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/downgrade/catdwgrd*.log] files for output generated by scripts

catcon::set_log_file_base_path: catcon: See [/u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/downgrade/catdwgrd_*.lst] files for spool files, if any

Diagnostic output produced in conjunction with reporting an error message:
{

2026-06-17 02:21:46 DEBUG> /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catcon.pm:12490 catcon::catconInit2 - will retain last 1000 log messages by default

2026-06-17 02:21:46 DEBUG> /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catcon.pm:9744 catcon::get_log_file_base_path - log file directory = /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/downgrade

2026-06-17 02:21:46 DEBUG> /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catcon.pm:9758 catcon::get_log_file_base_path - log file base = catdwgrd
.
.
.
.
.
.
.

2026-06-17 02:21:49 DEBUG> /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catcon.pm:18322 catcon::catconRevertPdbModes - catcon_RevertSeedPdbMode was not set

2026-06-17 02:21:49 DEBUG> /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catcon.pm:18369 catcon::catconRevertPdbModes - catcon_RevertUserPdbModes was not set

2026-06-17 02:21:49 VERBOSE> /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catcon.pm:17904 catcon::catconWrapUp - done

2026-06-17 02:21:49 DEBUG> /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catcon.pm:9914 catcon::next_proc - sending SIGKILL to SQL*Plus processes

2026-06-17 02:21:49 DEBUG> /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catcon.pm:9918 catcon::next_proc - returning -1 to the caller

}
End of diagnostic output

catcon::pickNextProc: unexpected error in next_proc

catcon::catconExec_int: unexpected error in pickNextProc

catcon::catconExec: set of Containers included the Root, but an error
        was encountered while processing PDBs so no
        attempt will be made to run it in CDB$ROOT

catcon::catconExec: an error (1) was encountered

catcon.pl: unexpected error encountered in catconExec
        while running scripts against Containers at UPGRADE_LEVEL of 1; exiting
catcon::catconWrapUp: catconInit2 has not been run

catcon.pl: completed with errors
[oracle@oel8-lab01 bin]$

You need to run catcon , catrelod with the below options , ensure your 19.19 OH is set to it's correct environment variables.


[oracle@oel8-lab01 19c]$ env | grep ORA
ORACLE_UNQNAME=cdb1
ORACLE_SID=cdb1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1  <<-- 19.19 OH
ORA_INVENTORY=/u01/app/oraInventory
ORACLE_HOSTNAME=oel8-lab01.nabhaas.com

Run the following command in screen , for an very small database this took 45 mins for me. This will basically reconfigure all your components in dba_regisry with 19.19

$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -e -n 1 -b catrelod -d $ORACLE_HOME/rdbms/admin catrelod.sql

Running catreload with the required arguments

$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -e -n 1 -b catrelod -d $ORACLE_HOME/rdbms/admin catrelod.sql



$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -e -n 1 -b catrelod -d $ORACLE_HOME/rdbms/admin catrelod.sql


[oracle@oel8-lab01 ~]$ tail -50f /u01/app/oracle/product/19.0.0/dbhome_1/bin/catrelod0.log 

.
.


Grant succeeded.

SQL> grant read on ku$_m_view_view to public
  2  /

Grant succeeded.

SQL> grant read on ku$_zm_view_view to public
  2  /

Grant succeeded.

SQL> grant read on ku$_m_view_view_base to public
  2  /

Grant succeeded.

SQL> grant read on ku$_m_view_h_view to public
  2  /

Grant succeeded.

SQL> grant read on ku$_zm_view_h_view to public
  2  /

Grant succeeded.

SQL> grant read on ku$_m_view_ph_view to public
  2  /

Grant succeeded.

SQL> grant read on ku$_zm_view_ph_view to public
  2  /

Grant succeeded.

SQL> grant read on ku$_m_view_fh_view to public
  2  /
^C

.
.
.

SQL> alter session set "_ORACLE_SCRIPT" = false;

Session altered.

SQL> 
SQL> 
SQL> 
SQL> Rem *******************************************************************
SQL> Rem END catrelod.sql
SQL> Rem *******************************************************************
SQL> 
END_RUNNING
--------------------------------------------------------------------------------
==== @/u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catrelod.sql Container
:PDB1 Id:3 26-06-17 02:44:48 Proc:0 ====


1 row selected.

SQL> 
END_RUNNING
--------------------------------------------------------------------------------
==== @/u01/app/oracle/product/19.0.0/dbhome_1/rdbms/admin/catrelod.sql Container
:PDB1 Id:3 26-06-17 02:44:48 Proc:0 ====


1 row selected.

SQL> SQL> 
SQL> SQL>   2  
Session altered.

SQL> SQL> ========== PROCESS ENDED ==========
SQL> ========== Process Terminated by catcon ==========
SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.19.0.0.0

Now checking dba_registry

cdb1> select * from dba_registry;

COMP_ID
------------------------------
COMP_NAME
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
VERSION                       |VERSION_FULL                  |STATUS         |MODIFIED                     |NAMESPACE
------------------------------|------------------------------|---------------|-----------------------------|------------------------------
CONTROL
--------------------------------------------------------------------------------------------------------------------------------
SCHEMA
--------------------------------------------------------------------------------------------------------------------------------
PROCEDURE                                                                                                                       |STARTUP |PARENT_ID
--------------------------------------------------------------------------------------------------------------------------------|--------|------------------------------
OTHER_SCHEMAS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CATALOG
Oracle Database Catalog Views
19.0.0.0.0                    |19.19.0.0.0                   |LOADED         |17-JUN-2026 02:24:24         |SERVER
SYS
SYS
DBMS_REGISTRY_SYS.VALIDATE_CATALOG                                                                                              |        |


CATPROC
Oracle Database Packages and Types
19.0.0.0.0                    |19.19.0.0.0                   |LOADED         |17-JUN-2026 02:28:28         |SERVER
SYS
SYS
DBMS_REGISTRY_SYS.VALIDATE_CATPROC                                                                                              |        |
APPQOSSYS,AUDSYS,DBSFWUSER,DBSNMP,DIP,GGSYS,GSMADMIN_INTERNAL,GSMCATUSER,GSMROOTUSER,GSMUSER,ORACLE_OCM,OUTLN,REMOTE_SCHEDULER_AGENT,SYS$UMF,SYSBACKUP,SYSDG,SYSKM,SYSRAC,SYSTEM

RAC
Oracle Real Application Clusters
19.0.0.0.0                    |19.19.0.0.0                   |OPTION OFF     |17-JUN-2026 02:29:02         |SERVER
SYS
SYS
DBMS_CLUSTDB.VALIDATE                                                                                                           |        |


JAVAVM
JServer JAVA Virtual Machine
19.0.0.0.0                    |19.31.0.0.0                   |VALID          |17-JUN-2026 00:47:31         |SERVER
SYS
SYS
INITJVMAUX.VALIDATE_JAVAVM                                                                                                      |        |
OJVMSYS

XML
Oracle XDK
19.0.0.0.0                    |19.19.0.0.0                   |VALID          |17-JUN-2026 02:28:37         |SERVER
SYS
SYS
XMLVALIDATE                                                                                                                     |        |


CATJAVA
Oracle Database Java Packages
19.0.0.0.0                    |19.19.0.0.0                   |LOADED         |17-JUN-2026 02:28:41         |SERVER
SYS
SYS
DBMS_REGISTRY_SYS.VALIDATE_CATJAVA                                                                                              |        |


APS
OLAP Analytic Workspace
19.0.0.0.0                    |19.19.0.0.0                   |VALID          |17-JUN-2026 02:28:45         |SERVER
SYS
SYS
APS_VALIDATE                                                                                                                    |        |


XDB
Oracle XML Database
19.0.0.0.0                    |19.19.0.0.0                   |VALID          |17-JUN-2026 02:29:19         |SERVER
SYS
XDB
DBMS_REGXDB.VALIDATEXDB                                                                                                         |        |
ANONYMOUS,XS$NULL

OWM
Oracle Workspace Manager
19.0.0.0.0                    |19.19.0.0.0                   |VALID          |17-JUN-2026 02:29:02         |SERVER
SYS
WMSYS
VALIDATE_OWM                                                                                                                    |        |


CONTEXT
Oracle Text
19.0.0.0.0                    |19.19.0.0.0                   |VALID          |17-JUN-2026 02:28:53         |SERVER
SYS
CTXSYS
VALIDATE_CONTEXT                                                                                                                |        |


ORDIM
Oracle Multimedia
19.0.0.0.0                    |19.19.0.0.0                   |VALID          |17-JUN-2026 02:29:28         |SERVER
SYS
ORDSYS
VALIDATE_ORDIM                                                                                                                  |        |
ORDDATA,ORDPLUGINS,SI_INFORMTN_SCHEMA

SDO
Spatial
19.0.0.0.0                    |19.19.0.0.0                   |INVALID        |17-JUN-2026 02:31:46         |SERVER
SYS
MDSYS
VALIDATE_SDO                                                                                                                    |        |
MDDATA,MDSYS

DV
Oracle Database Vault
19.0.0.0.0                    |19.19.0.0.0                   |INVALID        |17-JUN-2026 02:28:48         |SERVER
SYS
DVSYS
VALIDATE_DV                                                                                                                     |        |
DVF

XOQ
Oracle OLAP API
19.0.0.0.0                    |19.19.0.0.0                   |INVALID        |17-JUN-2026 02:31:50         |SERVER
SYS
OLAPSYS
XOQ_VALIDATE                                                                                                                    |        |


OLS
Oracle Label Security
19.0.0.0.0                    |19.19.0.0.0                   |INVALID        |17-JUN-2026 02:28:47         |SERVER
SYS
LBACSYS
VALIDATE_OLS                                                                                                                    |        |


15 rows selected.

Elapsed: 00:00:00.02
cdb1>

So this wraps up the rollback activity , cdb1 is back to running under /u01/app/oracle/product/19.0.0/dbhome_1

Learnings

Rollback is a very messy activity

The first take away is every organization have to define to a patching strategy and work backwards.

autoupgrade.jar patching in this scenario is out of place patching , so one needs to understand all the TNS entries , listener settings, sqlnet settings needs to be re-validated once moved to a new OH.
AutoUpgrade performs extensive pre-validation checks and will not proceed unless all requested patch components (RU, OJVM, DPBP, OPatch) are available and correctly identified.
Understanding the distinction between create_home, analyze, deploy, and restore phases is critical for successful execution.
AutoUpgrade maintains job state information and automatically resumes previously executed jobs, making it important to review existing job status before restarting an operation.
Oracle Home paths must be validated carefully; even minor path inconsistencies can prevent AutoUpgrade from recognizing a target Oracle Home.
The create_home functionality significantly simplifies out-of-place patching by automating Oracle Home creation, OPatch installation, inventory configuration, and RU application.
AutoUpgrade enforces version validation and prevents unsupported downgrade scenarios by ensuring the target Oracle Home is at the same or higher RU level than the source Oracle Home.
Verifying the newly created Oracle Home using opatch lspatches, inventory checks, and Oracle binaries before performing a database move is an essential best practice.
Out-of-place patching provides a safer patching strategy because the original Oracle Home remains untouched throughout the process.
Rollback procedures are significantly simplified when the original Oracle Home is preserved and available for fallback.
AutoUpgrade’s logging framework provides detailed diagnostics that greatly assist in troubleshooting validation, patching, and deployment issues.
Testing rollback procedures is as important as testing patch deployment procedures, ensuring operational readiness before production implementations.
AutoUpgrade is not merely an upgrade tool; it serves as a comprehensive automation framework for Oracle Home provisioning, patching, database upgrades, database moves, validation, and rollback management.
Proper planning, validation, and phased execution can significantly reduce patching risks and minimize manual intervention during Oracle Database maintenance activities.
Hands-on testing revealed that AutoUpgrade’s built-in safeguards help prevent many common human errors typically encountered during manual patching and Oracle Home migration exercises.

Reposting here about the main page for autoupgrade for understanding all it's use cases.
https://docs.oracle.com/en/database/oracle/oracle-database/19/upgrd/about-oracle-database-autoupgrade.html

OEL Linux iperf3 for Network Latency Monitoring

Abhilash Kumar | Oracle ACE ♠ — Mon, 15 Jun 2026 07:48:17 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

OEL Linux has built in network monitoring tool to monitor latency ,
I found this tool very useful and handy while working on Om Prem to cloud migrations to understand the network throughput.

The official Oracle Linux Documentation for iperf3 is here
https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/networkperformance.htm

In case iperf3 is not installed in your OEL , the below can be used , it basically runs on 5201 TCP ports

sudo yum install -y iperf3

Oracle recommends the use of iPerf3, an industry-standard network testing utility, to measure throughput between compute instances within OCI. By establishing a client-server connection between two Linux hosts, administrators can accurately benchmark available bandwidth, identify network constraints, and validate that their infrastructure is delivering the expected performance levels. Oracle’s network performance methodology uses iPerf3 to measure throughput between instances and compare the results against OCI’s documented network capabilities and service-level objectives.

iperf3 as a server mode

If you would like to collect network performance metrics on the server side , the following can be user

[root@instance-20260614-1905 ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.10.3.88, port 56520
[  5] local 10.10.3.88 port 5201 connected to 10.10.3.88 port 56528
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  7.40 GBytes  63.4 Gbits/sec                  
[  5]   1.00-2.00   sec  7.48 GBytes  64.3 Gbits/sec                  
[  5]   2.00-3.00   sec  7.88 GBytes  67.7 Gbits/sec                  
[  5]   3.00-4.00   sec  7.86 GBytes  67.5 Gbits/sec                  
[  5]   4.00-5.00   sec  7.77 GBytes  66.6 Gbits/sec                  
[  5]   5.00-6.00   sec  7.40 GBytes  63.7 Gbits/sec                  
[  5]   6.00-7.00   sec  7.03 GBytes  60.3 Gbits/sec                  
[  5]   7.00-8.00   sec  7.87 GBytes  67.7 Gbits/sec                  
[  5]   8.00-9.00   sec  7.95 GBytes  68.3 Gbits/sec                  
[  5]   9.00-10.00  sec  7.89 GBytes  67.9 Gbits/sec                  
[  5]  10.00-10.00  sec  8.38 MBytes  53.0 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  76.5 GBytes  65.7 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------

iperf3 for remote host network monitoring

I found the remote betwork monitoring very useful while data is in transit

Syntax :

iperf3 -c <your remote IP>

Example :

[opc@instance-20260614-1905 ~]$ iperf3 -c 10.10.3.88
Connecting to host 10.10.3.88, port 5201
[  5] local 10.10.3.88 port 56528 connected to 10.10.3.88 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.40 GBytes  63.4 Gbits/sec   54   2.69 MBytes       
[  5]   1.00-2.00   sec  7.48 GBytes  64.3 Gbits/sec   32   2.81 MBytes       
[  5]   2.00-3.00   sec  7.88 GBytes  67.7 Gbits/sec   18   2.81 MBytes       
[  5]   3.00-4.00   sec  7.86 GBytes  67.5 Gbits/sec    4   2.81 MBytes       
[  5]   4.00-5.00   sec  7.76 GBytes  66.7 Gbits/sec   19   2.12 MBytes       
[  5]   5.00-6.00   sec  6.92 GBytes  59.4 Gbits/sec   18   1.69 MBytes       
[  5]   6.00-7.00   sec  7.01 GBytes  60.3 Gbits/sec   90   2.12 MBytes       
[  5]   7.00-8.00   sec  7.66 GBytes  65.6 Gbits/sec    5   2.25 MBytes       
[  5]   8.00-9.00   sec  7.93 GBytes  68.3 Gbits/sec    0   2.25 MBytes       
[  5]   9.00-10.00  sec  7.81 GBytes  67.0 Gbits/sec    3   2.00 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  76.5 GBytes  65.7 Gbits/sec  243             sender
[  5]   0.00-10.00  sec  76.5 GBytes  65.7 Gbits/sec                  receiver

iperf Done.
[opc@instance-20260614-1905 ~]$

perf-check.py - Advanced Usage with MOS (My Oracle Support)

iperf3 is one the most regularly used commands Oracle Support uses for troubleshooting , there is also an advance version of it in the
forrm of pythin scripts for its usage.

https://docs.oracle.com/iaas/Content/Resources/Assets/perf-check.zip

root@instance-20260614-1905 script]# python perf-check.py --help
/root/scripts/script/perf-check.py:244: SyntaxWarning: invalid escape sequence '\.'
  version_regex = re.search("[0-9]\.[0-9]\.[0-9]", iperf_version)
usage: perf-check.py [-h] [--bind ADDRESS] [--output-file FILE] [--iperf IPERF] [--include-metadata] {client,server} ...

OCI Network Performance Analyzer

This tool assists with gathering network performance measurements
between OCI instances by running 'iperf' captures.  Run the tool
on two instances (i.e. 'client' and 'server') to perform a capture.

The 'iperf3' tool must be installed and tcp/udp port 5201 must be open.

positional arguments:
  {client,server}
    client            Start performance test in client mode
    server            Start performance test in server mode

options:
  -h, --help          show this help message and exit
  --bind ADDRESS      Bind test to given host address
  --output-file FILE  Archive to store test results
  --iperf IPERF       Path to iperf executable
  --include-metadata  Collect system metadata

To perform a capture:
Start the tool in 'server' mode:
    perf-check.py server
On another OCI instance. Start a 'client':
    perf-check.py client <server address>
Provide generated capture archives to OCI Support upon completion.
    Defaults: perf-results-client.tar.gz, perf-results-server.tar.gz
[root@instance-20260614-1905 script]#

perf-check.py - Server Mode

[root@instance-20260614-1905 script]# python perf-check.py server
/root/scripts/script/perf-check.py:244: SyntaxWarning: invalid escape sequence '\.'
  version_regex = re.search("[0-9]\.[0-9]\.[0-9]", iperf_version)
Running test in server mode...
/root/scripts/script/perf-check.py:246: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  curr_ver = LooseVersion(version_regex.group(0))
/root/scripts/script/perf-check.py:247: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  min_ver = LooseVersion("3.3.0")

###############################################################
Installed iperf version (1.7.1) is lower than recommended (3.3.0)

UDP test results may be unreliable
###############################################################

Beginning iperf TCP test...

TCP Performance Results:
iperf 3.17.1
Linux instance-20260614-1905 6.12.0-203.76.7.3.el10uek.x86_64 #2 SMP PREEMPT_DYNAMIC Fri Jun  5 00:17:00 PDT 2026 x86_64



Beginning iperf UDP test...

UDP Performance Results:
iperf 3.17.1
Linux instance-20260614-1905 6.12.0-203.76.7.3.el10uek.x86_64 #2 SMP PREEMPT_DYNAMIC Fri Jun  5 00:17:00 PDT 2026 x86_64



Performance test completed.  Please provide the 'perf-results-server.tar.gz' archive to Oracle Support.

perf-check.py - Client Mode - Probably the most useful usecase for this tool

[root@instance-20260614-1905 script]# python perf-check.py client 10.10.3.88
/root/scripts/script/perf-check.py:244: SyntaxWarning: invalid escape sequence '\.'
  version_regex = re.search("[0-9]\.[0-9]\.[0-9]", iperf_version)
Running test in client mode...
/root/scripts/script/perf-check.py:246: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  curr_ver = LooseVersion(version_regex.group(0))
/root/scripts/script/perf-check.py:247: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  min_ver = LooseVersion("3.3.0")

###############################################################
Installed iperf version (1.7.1) is lower than recommended (3.3.0)

UDP test results may be unreliable
###############################################################

Beginning TCP performance test...


TCP performance results:
iperf 3.17.1
Linux instance-20260614-1905 6.12.0-203.76.7.3.el10uek.x86_64 #2 SMP PREEMPT_DYNAMIC Fri Jun  5 00:17:00 PDT 2026 x86_64
Control connection MSS 32768
Time: Mon, 15 Jun 2026 07:31:51 GMT
Connecting to host 10.10.3.88, port 5201
      Cookie: cnedljaqiiih7ldlv3hfe46nlazengnyxo2a
      TCP MSS: 32768 (default)
[  5] local 10.10.3.88 port 51720 connected to 10.10.3.88 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.10 GBytes  60.9 Gbits/sec   11   2.06 MBytes       
[  5]   1.00-2.00   sec  6.89 GBytes  59.1 Gbits/sec   73   2.44 MBytes       
[  5]   2.00-3.00   sec  7.41 GBytes  63.7 Gbits/sec    8   2.87 MBytes       
[  5]   3.00-4.00   sec  7.79 GBytes  67.0 Gbits/sec    6   2.12 MBytes       
[  5]   4.00-5.00   sec  7.87 GBytes  67.6 Gbits/sec    0   2.12 MBytes       
[  5]   5.00-6.00   sec  7.88 GBytes  67.7 Gbits/sec    0   2.37 MBytes       
[  5]   6.00-7.00   sec  7.00 GBytes  60.1 Gbits/sec    5   2.94 MBytes       
[  5]   7.00-8.00   sec  7.83 GBytes  67.3 Gbits/sec    0   3.25 MBytes       
[  5]   8.00-9.00   sec  7.95 GBytes  68.3 Gbits/sec    0   3.37 MBytes       
[  5]   9.00-10.00  sec  7.83 GBytes  67.2 Gbits/sec    0   3.37 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  76.5 GBytes  65.7 Gbits/sec  103             sender
[  5]   0.00-10.00  sec  76.5 GBytes  65.7 Gbits/sec                  receiver
CPU Utilization: local/sender 78.9% (1.0%u/77.9%s), remote/receiver 74.9% (1.9%u/73.0%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.



Beginning UDP performance test...

UDP performance results:
iperf 3.17.1
Linux instance-20260614-1905 6.12.0-203.76.7.3.el10uek.x86_64 #2 SMP PREEMPT_DYNAMIC Fri Jun  5 00:17:00 PDT 2026 x86_64
Control connection MSS 32768
Setting UDP block size to 32768
Time: Mon, 15 Jun 2026 07:32:07 GMT
Connecting to host 10.10.3.88, port 5201
      Cookie: o6e2xygr3e3w2aee5v4m6k2oq7rjmvqb7jgf
      Target Bitrate: 2000000000
[  5] local 10.10.3.88 port 53831 connected to 10.10.3.88 port 5201
Starting Test: protocol: UDP, 1 streams, 32768 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   238 MBytes  2.00 Gbits/sec  7623  
[  5]   1.00-2.00   sec   238 MBytes  2.00 Gbits/sec  7623  
[  5]   2.00-3.00   sec   239 MBytes  2.00 Gbits/sec  7636  
[  5]   3.00-4.00   sec   238 MBytes  2.00 Gbits/sec  7629  
[  5]   4.00-5.00   sec   238 MBytes  2.00 Gbits/sec  7630  
[  5]   5.00-6.00   sec   238 MBytes  2.00 Gbits/sec  7629  
[  5]   6.00-7.00   sec   238 MBytes  2.00 Gbits/sec  7629  
[  5]   7.00-8.00   sec   238 MBytes  2.00 Gbits/sec  7630  
[  5]   8.00-9.00   sec   238 MBytes  2.00 Gbits/sec  7629  
[  5]   9.00-10.00  sec   238 MBytes  2.00 Gbits/sec  7630  
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec  0.000 ms  0/76288 (0%)  sender
[  5]   0.00-10.00  sec  2.07 GBytes  1.78 Gbits/sec  0.041 ms  8373/76288 (11%)  receiver
CPU Utilization: local/sender 95.5% (94.3%u/1.2%s), remote/receiver 41.9% (7.4%u/34.6%s)

iperf Done.



Performance test completed.  Please provide the 'perf-results-client.tar.gz' archive to Oracle Support.
[root@instance-20260614-1905 script]#

perf-check.py - Logs

You could see the logs generated as below

[root@instance-20260614-1905 script]# ls -l
total 32
-rw-r--r--. 1 root root  1828 Jun 14  2021 LICENSE.txt
-rwxr-xr-x. 1 root root 14296 Jun 14  2021 perf-check.py
-rw-r--r--. 1 root root  5497 Jun 15 07:32 perf-results-client.tar.gz
-rw-r--r--. 1 root root  3820 Jun 15 07:31 perf-results-server.tar.gz
[root@instance-20260614-1905 script]#

Understanding perf-check.py client metrics

A typical example of perf-check.py client metrics is shown below , this super useful for understanding latency asssociated with TCP / UDP.

[root@instance-20260614-1905 perf-results-client]# ls -l
total 84
-rw-r--r--. 1 root root  1797 Jun 15 07:32 iperf3_tcp.out
-rw-r--r--. 1 root root  1651 Jun 15 07:32 iperf3_udp.out
-rw-r--r--. 1 root root   368 Jun 15 07:31 iperf3_version.out
-rw-r--r--. 1 root root   432 Jun 15 07:31 ping.out
-rw-r--r--. 1 root root 15475 Jun 15 07:32 top_posttcp.out
-rw-r--r--. 1 root root 15475 Jun 15 07:32 top_postudp.out
-rw-r--r--. 1 root root 15475 Jun 15 07:31 top_pretcp.out
-rw-r--r--. 1 root root 15475 Jun 15 07:32 top_preudp.out
-rw-r--r--. 1 root root   139 Jun 15 07:32 tracepath.out

plaintext

An example of all perf-results-client is below , this is what typically MOS support logs would need to troubleshoot.

[root@instance-20260614-1905 perf-results-client]# cat *.out
iperf 3.17.1
Linux instance-20260614-1905 6.12.0-203.76.7.3.el10uek.x86_64 #2 SMP PREEMPT_DYNAMIC Fri Jun  5 00:17:00 PDT 2026 x86_64
Control connection MSS 32768
Time: Mon, 15 Jun 2026 07:31:51 GMT
Connecting to host 10.10.3.88, port 5201
      Cookie: cnedljaqiiih7ldlv3hfe46nlazengnyxo2a
      TCP MSS: 32768 (default)
[  5] local 10.10.3.88 port 51720 connected to 10.10.3.88 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  7.10 GBytes  60.9 Gbits/sec   11   2.06 MBytes       
[  5]   1.00-2.00   sec  6.89 GBytes  59.1 Gbits/sec   73   2.44 MBytes       
[  5]   2.00-3.00   sec  7.41 GBytes  63.7 Gbits/sec    8   2.87 MBytes       
[  5]   3.00-4.00   sec  7.79 GBytes  67.0 Gbits/sec    6   2.12 MBytes       
[  5]   4.00-5.00   sec  7.87 GBytes  67.6 Gbits/sec    0   2.12 MBytes       
[  5]   5.00-6.00   sec  7.88 GBytes  67.7 Gbits/sec    0   2.37 MBytes       
[  5]   6.00-7.00   sec  7.00 GBytes  60.1 Gbits/sec    5   2.94 MBytes       
[  5]   7.00-8.00   sec  7.83 GBytes  67.3 Gbits/sec    0   3.25 MBytes       
[  5]   8.00-9.00   sec  7.95 GBytes  68.3 Gbits/sec    0   3.37 MBytes       
[  5]   9.00-10.00  sec  7.83 GBytes  67.2 Gbits/sec    0   3.37 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  76.5 GBytes  65.7 Gbits/sec  103             sender
[  5]   0.00-10.00  sec  76.5 GBytes  65.7 Gbits/sec                  receiver
CPU Utilization: local/sender 78.9% (1.0%u/77.9%s), remote/receiver 74.9% (1.9%u/73.0%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.
iperf 3.17.1
Linux instance-20260614-1905 6.12.0-203.76.7.3.el10uek.x86_64 #2 SMP PREEMPT_DYNAMIC Fri Jun  5 00:17:00 PDT 2026 x86_64
Control connection MSS 32768
Setting UDP block size to 32768
Time: Mon, 15 Jun 2026 07:32:07 GMT
Connecting to host 10.10.3.88, port 5201
      Cookie: o6e2xygr3e3w2aee5v4m6k2oq7rjmvqb7jgf
      Target Bitrate: 2000000000
[  5] local 10.10.3.88 port 53831 connected to 10.10.3.88 port 5201
Starting Test: protocol: UDP, 1 streams, 32768 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   238 MBytes  2.00 Gbits/sec  7623  
[  5]   1.00-2.00   sec   238 MBytes  2.00 Gbits/sec  7623  
[  5]   2.00-3.00   sec   239 MBytes  2.00 Gbits/sec  7636  
[  5]   3.00-4.00   sec   238 MBytes  2.00 Gbits/sec  7629  
[  5]   4.00-5.00   sec   238 MBytes  2.00 Gbits/sec  7630  
[  5]   5.00-6.00   sec   238 MBytes  2.00 Gbits/sec  7629  
[  5]   6.00-7.00   sec   238 MBytes  2.00 Gbits/sec  7629  
[  5]   7.00-8.00   sec   238 MBytes  2.00 Gbits/sec  7630  
[  5]   8.00-9.00   sec   238 MBytes  2.00 Gbits/sec  7629  
[  5]   9.00-10.00  sec   238 MBytes  2.00 Gbits/sec  7630  
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec  0.000 ms  0/76288 (0%)  sender
[  5]   0.00-10.00  sec  2.07 GBytes  1.78 Gbits/sec  0.041 ms  8373/76288 (11%)  receiver
CPU Utilization: local/sender 95.5% (94.3%u/1.2%s), remote/receiver 41.9% (7.4%u/34.6%s)

iperf Done.
iperf 3.17.1 (cJSON 1.7.15)
Linux instance-20260614-1905 6.12.0-203.76.7.3.el10uek.x86_64 #2 SMP PREEMPT_DYNAMIC Fri Jun  5 00:17:00 PDT 2026 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threads
PING 10.10.3.88 (10.10.3.88) 56(84) bytes of data.
64 bytes from 10.10.3.88: icmp_seq=1 ttl=64 time=0.020 ms
64 bytes from 10.10.3.88: icmp_seq=2 ttl=64 time=0.021 ms
64 bytes from 10.10.3.88: icmp_seq=3 ttl=64 time=0.026 ms
64 bytes from 10.10.3.88: icmp_seq=4 ttl=64 time=0.043 ms

--- 10.10.3.88 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3077ms
rtt min/avg/max/mdev = 0.020/0.027/0.043/0.009 ms
top - 07:32:02 up 8 min,  5 users,  load average: 0.31, 0.12, 0.05
Tasks: 189 total,   1 running, 188 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  4.5 sy,  0.0 ni, 95.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   9480.8 total,   8338.1 free,    695.9 used,    711.9 buff/cache     
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.   8785.0 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0   49620  41764  10832 S   0.0   0.4   0:02.53 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_wo+
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      9 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
     10 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     11 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
     13 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     14 root      20   0       0      0      0 I   0.0   0.0   0:00.60 kworker+
     15 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     16 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     17 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     18 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftir+
     19 root      20   0       0      0      0 I   0.0   0.0   0:00.06 rcu_pre+
     20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_exp+
     21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_exp+
     22 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migrati+
     23 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     24 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     25 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
     26 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     27 root      rt   0       0      0      0 S   0.0   0.0   0:00.17 migrati+
     28 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftir+
     29 root      20   0       0      0      0 I   0.0   0.0   0:00.01 kworker+
     30 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     32 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kdevtmp+
     33 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     34 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kauditd
     35 root      20   0       0      0      0 S   0.0   0.0   0:00.00 khungta+
     36 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
     37 root      20   0       0      0      0 S   0.0   0.0   0:00.00 oom_rea+
     38 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     39 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kcompac+
     40 root      25   5       0      0      0 S   0.0   0.0   0:00.00 ksmd
     41 root      39  19       0      0      0 S   0.0   0.0   0:00.05 khugepa+
     42 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     43 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     44 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     45 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     46 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 irq/9-a+
     47 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
     48 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     49 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     50 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     51 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     52 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     53 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 watchdo+
     54 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     55 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     56 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kswapd0
     74 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     79 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     80 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     81 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     82 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     83 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     84 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     85 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     86 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     87 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     88 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     89 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     95 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    130 root      20   0       0      0      0 I   0.0   0.0   0:00.04 kworker+
    138 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    231 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    232 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
    233 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    234 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    235 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    281 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    283 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    418 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    422 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    424 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    425 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 bnx2i_t+
    426 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 bnx2i_t+
    511 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    512 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    638 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    639 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    648 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    658 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    659 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    660 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    666 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    724 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    725 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    726 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    727 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    728 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1059 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1068 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1097 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1098 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1099 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1100 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1101 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1102 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1103 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1104 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1105 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1106 root      20   0       0      0      0 S   0.0   0.0   0:00.05 xfsaild+
   1222 root      20   0   25364  10208   8828 S   0.0   0.1   0:00.15 systemd+
   1275 root      20   0   15860   6228   5248 S   0.0   0.1   0:00.02 systemd+
   1287 root      20   0   35372  12648   8652 S   0.0   0.1   0:00.07 systemd+
   1549 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1550 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1551 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1552 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1553 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1555 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1557 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1559 root      20   0       0      0      0 S   0.0   0.0   0:00.00 xfsaild+
   1599 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1662 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1666 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1667 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1668 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1669 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1670 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1671 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1672 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1673 root      20   0       0      0      0 S   0.0   0.0   0:00.00 xfsaild+
   1719 rpc       20   0   12624   4920   4004 S   0.0   0.1   0:00.01 rpcbind
   1720 root      16  -4   20440   2912   1952 S   0.0   0.0   0:00.01 auditd
   1749 dbus      20   0    8812   4224   3604 S   0.0   0.0   0:00.01 dbus-br+
   1750 root      20   0    4836   2348   2156 S   0.0   0.0   0:00.00 dtprobed
   1758 root      20   0 1052380    896    708 S   0.0   0.0   0:00.00 dtprobed
   1760 dbus      20   0    4912   2984   2472 S   0.0   0.0   0:00.09 dbus-br+
   1764 root      20   0   80408   3756   3432 S   0.0   0.0   0:00.03 irqbala+
   1766 root      20   0   18880  10136   7660 S   0.0   0.1   0:00.10 systemd+
   1804 root      20   0  330264  18248  15364 S   0.0   0.2   0:00.06 Network+
   1816 root      20   0   57140   7700   6524 S   0.0   0.1   0:00.02 gssproxy
   1817 root      20   0  256604  27680  13112 S   0.0   0.3   0:00.18 tuned
   1916 polkitd   20   0  305736   7404   6552 S   0.0   0.1   0:00.06 polkitd
   2194 root      10 -10   13852  13836  12004 S   0.0   0.1   0:00.00 iscsid
   2223 oracle-+  20   0 1241748  24116  13376 S   0.0   0.2   0:00.32 updater
   2235 root      20   0  163276   7392   5908 S   0.0   0.1   0:00.11 rsyslogd
   2241 root      20   0    8760   6524   5348 S   0.0   0.1   0:00.00 sshd
   2250 root      20   0  535832  45340  13392 S   0.0   0.5   0:00.68 fluentd
   2255 oracle-+  20   0 1241480  25716  15412 S   0.0   0.3   0:00.35 agent
   2300 chrony    20   0   10400   2984   2392 S   0.0   0.0   0:00.00 chronyd
   2351 root      20   0    8296   3472   2596 S   0.0   0.0   0:00.00 crond
   2353 root      20   0    4980   2344   2108 S   0.0   0.0   0:00.00 agetty
   2355 root      20   0    7548   2616   2360 S   0.0   0.0   0:00.00 agetty
   2365 oracle-+  20   0 1241292  25904  15344 S   0.0   0.3   0:00.31 gomon
   2415 root      20   0   18544   7936   6460 S   0.0   0.1   0:00.01 sudo
   2445 ocarun    20   0   22240  13464  10324 S   0.0   0.1   0:00.11 systemd
   2456 ocarun    20   0   14432   2904   1548 S   0.0   0.0   0:00.00 (sd-pam)
   2530 ocarun    20   0 1245024  28932  18520 S   0.0   0.3   0:00.18 runcomm+
   2537 pcp       20   0   14012   7688   5820 S   0.0   0.1   0:00.17 pmcd
   2551 root      20   0   13900   5096   4132 S   0.0   0.1   0:00.00 pmdaroot
   2566 root      20   0   14308   6448   4420 S   0.0   0.1   0:00.18 pmdaproc
   2579 root      20   0   13116   4828   3908 S   0.0   0.0   0:00.00 pmdaxfs
   2580 root      20   0   14032   6348   4668 S   0.0   0.1   0:00.15 pmdalin+
   2581 oracle-+  20   0 1241956  26044  15824 S   0.0   0.3   0:00.60 unified+
   2585 root      20   0   31304  23008  11324 S   0.0   0.2   0:00.05 python3
   2639 root      20   0   13096   4812   3896 S   0.0   0.0   0:00.00 pmdakvm
   2642 root      20   0   14636   5584   4544 S   0.0   0.1   0:00.00 pmdadm
   2647 pcp       20   0  262712  33464  13272 S   0.0   0.3   0:00.14 python3
   3356 pcp       20   0   15600   8292   5300 S   0.0   0.1   0:00.13 pmlogger
   3365 pcp       20   0   13916   6584   5328 S   0.0   0.1   0:00.00 pmie
   3381 pcp       20   0   12808   4160   3304 S   0.0   0.0   0:00.00 pmpause
   3493 pcp       20   0   12808   4168   3308 S   0.0   0.0   0:00.00 pmpause
   3957 root      20   0  526092  39020  13264 S   0.0   0.4   0:00.48 ruby
  10377 root      20   0   15496  10656   8632 S   0.0   0.1   0:00.00 sshd-se+
  10382 opc       20   0   22248  13480  10352 S   0.0   0.1   0:00.06 systemd
  10384 root      20   0   16612   7336   6260 S   0.0   0.1   0:00.00 systemd+
  10385 opc       20   0   14432   2912   1560 S   0.0   0.0   0:00.00 (sd-pam)
  10394 opc       20   0   15884   8120   5800 S   0.0   0.1   0:00.57 sshd-se+
  10395 opc       20   0    7212   6176   5496 S   0.0   0.1   0:00.00 bash
  10425 root      20   0   18552   9944   8460 S   0.0   0.1   0:00.03 sudo
  10427 root      20   0   16612   7464   6384 S   0.0   0.1   0:00.00 systemd+
  10428 root      20   0   18552   2832   1352 S   0.0   0.0   0:00.00 sudo
  10429 root      20   0   10088   7140   6324 S   0.0   0.1   0:00.00 su
  10430 root      20   0    7328   6220   5508 S   0.0   0.1   0:00.03 bash
  10512 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
  10721 root      20   0   15628  10648   8624 S   0.0   0.1   0:00.01 sshd-se+
  10724 root      20   0   16448   6920   5888 S   0.0   0.1   0:00.00 systemd+
  10725 opc       20   0   15888   8120   5800 S   0.0   0.1   0:00.00 sshd-se+
  10726 opc       20   0    7212   6156   5476 S   0.0   0.1   0:00.00 bash
  10756 opc       20   0   18036   3768   3048 S   0.0   0.0   0:07.49 iperf3
  10784 root      20   0   31752  26324   9788 S   0.0   0.3   0:00.10 python
  10798 root      20   0   10172   7088   4992 R   0.0   0.1   0:00.00 top
top - 07:32:17 up 8 min,  5 users,  load average: 0.86, 0.24, 0.09
Tasks: 189 total,   1 running, 188 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   9480.8 total,   8338.7 free,    695.1 used,    712.0 buff/cache     
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.   8785.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0   49620  41764  10832 S   0.0   0.4   0:02.53 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_wo+
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      9 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
     10 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     11 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
     13 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     14 root      20   0       0      0      0 I   0.0   0.0   0:00.60 kworker+
     15 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     16 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     17 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     18 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftir+
     19 root      20   0       0      0      0 I   0.0   0.0   0:00.06 rcu_pre+
     20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_exp+
     21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_exp+
     22 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migrati+
     23 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     24 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     25 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
     26 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     27 root      rt   0       0      0      0 S   0.0   0.0   0:00.17 migrati+
     28 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftir+
     29 root      20   0       0      0      0 I   0.0   0.0   0:00.01 kworker+
     30 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     32 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kdevtmp+
     33 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     34 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kauditd
     35 root      20   0       0      0      0 S   0.0   0.0   0:00.00 khungta+
     36 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
     37 root      20   0       0      0      0 S   0.0   0.0   0:00.00 oom_rea+
     38 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     39 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kcompac+
     40 root      25   5       0      0      0 S   0.0   0.0   0:00.00 ksmd
     41 root      39  19       0      0      0 S   0.0   0.0   0:00.05 khugepa+
     42 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     43 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     44 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     45 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     46 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 irq/9-a+
     47 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
     48 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     49 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     50 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     51 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     52 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     53 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 watchdo+
     54 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     55 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     56 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kswapd0
     74 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     79 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     80 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     81 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     82 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     83 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     84 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     85 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     86 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     87 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     88 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     89 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     95 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    130 root      20   0       0      0      0 I   0.0   0.0   0:00.04 kworker+
    138 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    231 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    232 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
    233 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    234 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    235 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    281 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    283 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    418 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    422 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    424 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    425 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 bnx2i_t+
    426 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 bnx2i_t+
    511 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    512 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    638 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    639 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    648 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    658 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    659 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    660 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    666 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    724 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    725 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    726 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    727 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    728 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1059 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1068 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1097 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1098 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1099 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1100 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1101 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1102 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1103 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1104 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1105 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1106 root      20   0       0      0      0 S   0.0   0.0   0:00.05 xfsaild+
   1222 root      20   0   25364  10208   8828 S   0.0   0.1   0:00.15 systemd+
   1275 root      20   0   15860   6228   5248 S   0.0   0.1   0:00.02 systemd+
   1287 root      20   0   35372  12648   8652 S   0.0   0.1   0:00.07 systemd+
   1549 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1550 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1551 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1552 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1553 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1555 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1557 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1559 root      20   0       0      0      0 S   0.0   0.0   0:00.00 xfsaild+
   1599 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1662 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1666 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1667 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1668 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1669 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1670 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1671 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1672 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1673 root      20   0       0      0      0 S   0.0   0.0   0:00.00 xfsaild+
   1719 rpc       20   0   12624   4920   4004 S   0.0   0.1   0:00.01 rpcbind
   1720 root      16  -4   20440   2912   1952 S   0.0   0.0   0:00.01 auditd
   1749 dbus      20   0    8812   4224   3604 S   0.0   0.0   0:00.01 dbus-br+
   1750 root      20   0    4836   2348   2156 S   0.0   0.0   0:00.00 dtprobed
   1758 root      20   0 1052380    896    708 S   0.0   0.0   0:00.00 dtprobed
   1760 dbus      20   0    4912   2984   2472 S   0.0   0.0   0:00.09 dbus-br+
   1764 root      20   0   80408   3756   3432 S   0.0   0.0   0:00.03 irqbala+
   1766 root      20   0   18880  10136   7660 S   0.0   0.1   0:00.10 systemd+
   1804 root      20   0  330264  18248  15364 S   0.0   0.2   0:00.06 Network+
   1816 root      20   0   57140   7700   6524 S   0.0   0.1   0:00.02 gssproxy
   1817 root      20   0  256604  27680  13112 S   0.0   0.3   0:00.19 tuned
   1916 polkitd   20   0  305736   7404   6552 S   0.0   0.1   0:00.06 polkitd
   2194 root      10 -10   13852  13836  12004 S   0.0   0.1   0:00.00 iscsid
   2223 oracle-+  20   0 1241748  24116  13376 S   0.0   0.2   0:00.32 updater
   2235 root      20   0  163276   7392   5908 S   0.0   0.1   0:00.11 rsyslogd
   2241 root      20   0    8760   6524   5348 S   0.0   0.1   0:00.00 sshd
   2250 root      20   0  535832  45340  13392 S   0.0   0.5   0:00.68 fluentd
   2255 oracle-+  20   0 1241480  26164  15412 S   0.0   0.3   0:00.35 agent
   2300 chrony    20   0   10400   2984   2392 S   0.0   0.0   0:00.00 chronyd
   2351 root      20   0    8296   3472   2596 S   0.0   0.0   0:00.00 crond
   2353 root      20   0    4980   2344   2108 S   0.0   0.0   0:00.00 agetty
   2355 root      20   0    7548   2616   2360 S   0.0   0.0   0:00.00 agetty
   2365 oracle-+  20   0 1241292  25904  15344 S   0.0   0.3   0:00.31 gomon
   2415 root      20   0   18544   7936   6460 S   0.0   0.1   0:00.01 sudo
   2445 ocarun    20   0   22240  13464  10324 S   0.0   0.1   0:00.11 systemd
   2456 ocarun    20   0   14432   2904   1548 S   0.0   0.0   0:00.00 (sd-pam)
   2530 ocarun    20   0 1245024  28932  18520 S   0.0   0.3   0:00.18 runcomm+
   2537 pcp       20   0   14012   7688   5820 S   0.0   0.1   0:00.17 pmcd
   2551 root      20   0   13900   5096   4132 S   0.0   0.1   0:00.00 pmdaroot
   2566 root      20   0   14308   6448   4420 S   0.0   0.1   0:00.19 pmdaproc
   2579 root      20   0   13116   4828   3908 S   0.0   0.0   0:00.00 pmdaxfs
   2580 root      20   0   14032   6348   4668 S   0.0   0.1   0:00.15 pmdalin+
   2581 oracle-+  20   0 1241956  26044  15824 S   0.0   0.3   0:00.60 unified+
   2585 root      20   0   31304  23008  11324 S   0.0   0.2   0:00.05 python3
   2639 root      20   0   13096   4812   3896 S   0.0   0.0   0:00.00 pmdakvm
   2642 root      20   0   14636   5584   4544 S   0.0   0.1   0:00.00 pmdadm
   2647 pcp       20   0  262712  33464  13272 S   0.0   0.3   0:00.14 python3
   3356 pcp       20   0   15600   8292   5300 S   0.0   0.1   0:00.13 pmlogger
   3365 pcp       20   0   13916   6584   5328 S   0.0   0.1   0:00.00 pmie
   3381 pcp       20   0   12808   4160   3304 S   0.0   0.0   0:00.00 pmpause
   3493 pcp       20   0   12808   4168   3308 S   0.0   0.0   0:00.00 pmpause
   3957 root      20   0  526092  39020  13264 S   0.0   0.4   0:00.48 ruby
  10377 root      20   0   15496  10656   8632 S   0.0   0.1   0:00.00 sshd-se+
  10382 opc       20   0   22248  13480  10352 S   0.0   0.1   0:00.06 systemd
  10385 opc       20   0   14432   2912   1560 S   0.0   0.0   0:00.00 (sd-pam)
  10394 opc       20   0   15884   8120   5800 S   0.0   0.1   0:00.57 sshd-se+
  10395 opc       20   0    7212   6176   5496 S   0.0   0.1   0:00.00 bash
  10425 root      20   0   18552   9944   8460 S   0.0   0.1   0:00.03 sudo
  10428 root      20   0   18552   2832   1352 S   0.0   0.0   0:00.00 sudo
  10429 root      20   0   10088   7140   6324 S   0.0   0.1   0:00.00 su
  10430 root      20   0    7328   6220   5508 S   0.0   0.1   0:00.03 bash
  10512 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
  10721 root      20   0   15628  10648   8624 S   0.0   0.1   0:00.01 sshd-se+
  10724 root      20   0   16448   6920   5888 S   0.0   0.1   0:00.00 systemd+
  10725 opc       20   0   15888   8120   5800 S   0.0   0.1   0:00.00 sshd-se+
  10726 opc       20   0    7212   6156   5476 S   0.0   0.1   0:00.00 bash
  10756 opc       20   0   18216   4052   3316 S   0.0   0.0   0:11.67 iperf3
  10784 root      20   0   31896  26416   9788 S   0.0   0.3   0:00.10 python
  10803 root      20   0   16304   6188   5184 S   0.0   0.1   0:00.00 systemd+
  10804 root      20   0   16304   6188   5180 S   0.0   0.1   0:00.00 systemd+
  10805 root      20   0   10172   7208   5116 R   0.0   0.1   0:00.00 top
top - 07:31:51 up 8 min,  5 users,  load average: 0.01, 0.05, 0.02
Tasks: 189 total,   1 running, 188 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   9480.8 total,   8313.6 free,    720.3 used,    711.9 buff/cache     
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.   8760.5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0   49620  41764  10832 S   0.0   0.4   0:02.53 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_wo+
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      9 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
     10 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     11 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
     13 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     14 root      20   0       0      0      0 I   0.0   0.0   0:00.60 kworker+
     15 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     16 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     17 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     18 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftir+
     19 root      20   0       0      0      0 I   0.0   0.0   0:00.06 rcu_pre+
     20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_exp+
     21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_exp+
     22 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migrati+
     23 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     24 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     25 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
     26 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     27 root      rt   0       0      0      0 S   0.0   0.0   0:00.17 migrati+
     28 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftir+
     29 root      20   0       0      0      0 I   0.0   0.0   0:00.01 kworker+
     30 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     32 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kdevtmp+
     33 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     34 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kauditd
     35 root      20   0       0      0      0 S   0.0   0.0   0:00.00 khungta+
     36 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
     37 root      20   0       0      0      0 S   0.0   0.0   0:00.00 oom_rea+
     38 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     39 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kcompac+
     40 root      25   5       0      0      0 S   0.0   0.0   0:00.00 ksmd
     41 root      39  19       0      0      0 S   0.0   0.0   0:00.05 khugepa+
     42 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     43 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     44 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     45 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     46 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 irq/9-a+
     47 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
     48 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     49 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     50 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     51 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     52 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     53 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 watchdo+
     54 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     55 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     56 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kswapd0
     74 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     79 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     80 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     81 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     82 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     83 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     84 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     85 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     86 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     87 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     88 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     89 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     95 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    130 root      20   0       0      0      0 I   0.0   0.0   0:00.04 kworker+
    138 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    231 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    232 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
    233 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    234 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    235 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    281 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    283 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    418 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    422 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    424 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    425 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 bnx2i_t+
    426 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 bnx2i_t+
    511 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    512 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    638 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    639 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    648 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    658 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    659 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    660 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    666 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    724 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    725 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    726 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    727 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    728 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1059 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1068 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1097 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1098 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1099 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1100 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1101 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1102 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1103 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1104 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1105 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1106 root      20   0       0      0      0 S   0.0   0.0   0:00.04 xfsaild+
   1222 root      20   0   25364  10208   8828 S   0.0   0.1   0:00.15 systemd+
   1275 root      20   0   15860   6228   5248 S   0.0   0.1   0:00.02 systemd+
   1287 root      20   0   35372  12648   8652 S   0.0   0.1   0:00.07 systemd+
   1549 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1550 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1551 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1552 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1553 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1555 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1557 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1559 root      20   0       0      0      0 S   0.0   0.0   0:00.00 xfsaild+
   1599 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1662 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1666 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1667 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1668 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1669 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1670 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1671 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1672 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1673 root      20   0       0      0      0 S   0.0   0.0   0:00.00 xfsaild+
   1719 rpc       20   0   12624   4920   4004 S   0.0   0.1   0:00.01 rpcbind
   1720 root      16  -4   20440   2912   1952 S   0.0   0.0   0:00.01 auditd
   1749 dbus      20   0    8812   4224   3604 S   0.0   0.0   0:00.01 dbus-br+
   1750 root      20   0    4836   2348   2156 S   0.0   0.0   0:00.00 dtprobed
   1758 root      20   0 1052380    896    708 S   0.0   0.0   0:00.00 dtprobed
   1760 dbus      20   0    4912   2984   2472 S   0.0   0.0   0:00.09 dbus-br+
   1764 root      20   0   80408   3756   3432 S   0.0   0.0   0:00.03 irqbala+
   1766 root      20   0   18880  10136   7660 S   0.0   0.1   0:00.10 systemd+
   1804 root      20   0  330264  18248  15364 S   0.0   0.2   0:00.06 Network+
   1816 root      20   0   57140   7700   6524 S   0.0   0.1   0:00.02 gssproxy
   1817 root      20   0  256604  27680  13112 S   0.0   0.3   0:00.18 tuned
   1916 polkitd   20   0  305736   7404   6552 S   0.0   0.1   0:00.06 polkitd
   2194 root      10 -10   13852  13836  12004 S   0.0   0.1   0:00.00 iscsid
   2223 oracle-+  20   0 1241748  24116  13376 S   0.0   0.2   0:00.32 updater
   2235 root      20   0  163276   7392   5908 S   0.0   0.1   0:00.11 rsyslogd
   2241 root      20   0    8760   6524   5348 S   0.0   0.1   0:00.00 sshd
   2250 root      20   0  535832  45340  13392 S   0.0   0.5   0:00.68 fluentd
   2255 oracle-+  20   0 1241480  25716  15412 S   0.0   0.3   0:00.35 agent
   2300 chrony    20   0   10400   2984   2392 S   0.0   0.0   0:00.00 chronyd
   2351 root      20   0    8296   3472   2596 S   0.0   0.0   0:00.00 crond
   2353 root      20   0    4980   2344   2108 S   0.0   0.0   0:00.00 agetty
   2355 root      20   0    7548   2616   2360 S   0.0   0.0   0:00.00 agetty
   2365 oracle-+  20   0 1241292  25904  15344 S   0.0   0.3   0:00.31 gomon
   2415 root      20   0   18544   7936   6460 S   0.0   0.1   0:00.01 sudo
   2445 ocarun    20   0   22240  13464  10324 S   0.0   0.1   0:00.11 systemd
   2456 ocarun    20   0   14432   2904   1548 S   0.0   0.0   0:00.00 (sd-pam)
   2530 ocarun    20   0 1245024  28932  18520 S   0.0   0.3   0:00.18 runcomm+
   2537 pcp       20   0   14012   7688   5820 S   0.0   0.1   0:00.17 pmcd
   2551 root      20   0   13900   5096   4132 S   0.0   0.1   0:00.00 pmdaroot
   2566 root      20   0   14308   6448   4420 S   0.0   0.1   0:00.18 pmdaproc
   2579 root      20   0   13116   4828   3908 S   0.0   0.0   0:00.00 pmdaxfs
   2580 root      20   0   14032   6348   4668 S   0.0   0.1   0:00.14 pmdalin+
   2581 oracle-+  20   0 1241956  26044  15824 S   0.0   0.3   0:00.60 unified+
   2585 root      20   0   31304  23008  11324 S   0.0   0.2   0:00.05 python3
   2639 root      20   0   13096   4812   3896 S   0.0   0.0   0:00.00 pmdakvm
   2642 root      20   0   14636   5584   4544 S   0.0   0.1   0:00.00 pmdadm
   2647 pcp       20   0  262712  33464  13272 S   0.0   0.3   0:00.14 python3
   3356 pcp       20   0   15600   8292   5300 S   0.0   0.1   0:00.13 pmlogger
   3365 pcp       20   0   13916   6584   5328 S   0.0   0.1   0:00.00 pmie
   3381 pcp       20   0   12808   4160   3304 S   0.0   0.0   0:00.00 pmpause
   3493 pcp       20   0   12808   4168   3308 S   0.0   0.0   0:00.00 pmpause
   3957 root      20   0  526092  39020  13264 S   0.0   0.4   0:00.48 ruby
  10377 root      20   0   15496  10656   8632 S   0.0   0.1   0:00.00 sshd-se+
  10382 opc       20   0   22248  13480  10352 S   0.0   0.1   0:00.06 systemd
  10384 root      20   0   16612   7336   6260 S   0.0   0.1   0:00.00 systemd+
  10385 opc       20   0   14432   2912   1560 S   0.0   0.0   0:00.00 (sd-pam)
  10394 opc       20   0   15884   8120   5800 S   0.0   0.1   0:00.57 sshd-se+
  10395 opc       20   0    7212   6176   5496 S   0.0   0.1   0:00.00 bash
  10425 root      20   0   18552   9944   8460 S   0.0   0.1   0:00.03 sudo
  10427 root      20   0   16612   7464   6384 S   0.0   0.1   0:00.00 systemd+
  10428 root      20   0   18552   2832   1352 S   0.0   0.0   0:00.00 sudo
  10429 root      20   0   10088   7140   6324 S   0.0   0.1   0:00.00 su
  10430 root      20   0    7328   6220   5508 S   0.0   0.1   0:00.03 bash
  10512 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
  10721 root      20   0   15628  10648   8624 S   0.0   0.1   0:00.01 sshd-se+
  10724 root      20   0   16448   6920   5888 S   0.0   0.1   0:00.00 systemd+
  10725 opc       20   0   15888   8120   5800 S   0.0   0.1   0:00.00 sshd-se+
  10726 opc       20   0    7212   6156   5476 S   0.0   0.1   0:00.00 bash
  10756 opc       20   0    9840   3616   2920 S   0.0   0.0   0:00.00 iperf3
  10784 root      20   0   31752  26308   9788 S   0.0   0.3   0:00.09 python
  10794 root      20   0   10172   7128   5036 R   0.0   0.1   0:00.00 top
top - 07:32:07 up 8 min,  5 users,  load average: 0.29, 0.12, 0.05
Tasks: 189 total,   1 running, 188 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   9480.8 total,   8337.0 free,    696.7 used,    712.0 buff/cache     
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.   8784.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0   49620  41764  10832 S   0.0   0.4   0:02.53 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_wo+
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      9 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
     10 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     11 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
     13 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     14 root      20   0       0      0      0 I   0.0   0.0   0:00.60 kworker+
     15 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     16 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     17 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tas+
     18 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftir+
     19 root      20   0       0      0      0 I   0.0   0.0   0:00.06 rcu_pre+
     20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_exp+
     21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_exp+
     22 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migrati+
     23 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     24 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     25 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
     26 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     27 root      rt   0       0      0      0 S   0.0   0.0   0:00.17 migrati+
     28 root      20   0       0      0      0 S   0.0   0.0   0:00.04 ksoftir+
     29 root      20   0       0      0      0 I   0.0   0.0   0:00.01 kworker+
     30 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     32 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kdevtmp+
     33 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     34 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kauditd
     35 root      20   0       0      0      0 S   0.0   0.0   0:00.00 khungta+
     36 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
     37 root      20   0       0      0      0 S   0.0   0.0   0:00.00 oom_rea+
     38 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     39 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kcompac+
     40 root      25   5       0      0      0 S   0.0   0.0   0:00.00 ksmd
     41 root      39  19       0      0      0 S   0.0   0.0   0:00.05 khugepa+
     42 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     43 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     44 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     45 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     46 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 irq/9-a+
     47 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
     48 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     49 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     50 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     51 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     52 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     53 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 watchdo+
     54 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     55 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     56 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kswapd0
     74 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     79 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     80 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     81 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     82 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     83 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     84 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     85 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     86 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     87 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     88 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     89 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
     95 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    130 root      20   0       0      0      0 I   0.0   0.0   0:00.04 kworker+
    138 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    231 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    232 root      20   0       0      0      0 I   0.0   0.0   0:00.03 kworker+
    233 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    234 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    235 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
    281 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    283 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    418 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    422 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    424 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    425 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 bnx2i_t+
    426 root       0 -20       0      0      0 S   0.0   0.0   0:00.00 bnx2i_t+
    511 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    512 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    638 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    639 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    648 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    658 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    659 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    660 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh+
    666 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    724 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    725 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    726 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    727 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
    728 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1059 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1068 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1097 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1098 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1099 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1100 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1101 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1102 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1103 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1104 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1105 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1106 root      20   0       0      0      0 S   0.0   0.0   0:00.05 xfsaild+
   1222 root      20   0   25364  10208   8828 S   0.0   0.1   0:00.15 systemd+
   1275 root      20   0   15860   6228   5248 S   0.0   0.1   0:00.02 systemd+
   1287 root      20   0   35372  12648   8652 S   0.0   0.1   0:00.07 systemd+
   1549 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1550 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1551 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1552 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1553 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1555 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1557 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1559 root      20   0       0      0      0 S   0.0   0.0   0:00.00 xfsaild+
   1599 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1662 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1666 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1667 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1668 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1669 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1670 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1671 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1672 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
   1673 root      20   0       0      0      0 S   0.0   0.0   0:00.00 xfsaild+
   1719 rpc       20   0   12624   4920   4004 S   0.0   0.1   0:00.01 rpcbind
   1720 root      16  -4   20440   2912   1952 S   0.0   0.0   0:00.01 auditd
   1749 dbus      20   0    8812   4224   3604 S   0.0   0.0   0:00.01 dbus-br+
   1750 root      20   0    4836   2348   2156 S   0.0   0.0   0:00.00 dtprobed
   1758 root      20   0 1052380    896    708 S   0.0   0.0   0:00.00 dtprobed
   1760 dbus      20   0    4912   2984   2472 S   0.0   0.0   0:00.09 dbus-br+
   1764 root      20   0   80408   3756   3432 S   0.0   0.0   0:00.03 irqbala+
   1766 root      20   0   18880  10136   7660 S   0.0   0.1   0:00.10 systemd+
   1804 root      20   0  330264  18248  15364 S   0.0   0.2   0:00.06 Network+
   1816 root      20   0   57140   7700   6524 S   0.0   0.1   0:00.02 gssproxy
   1817 root      20   0  256604  27680  13112 S   0.0   0.3   0:00.18 tuned
   1916 polkitd   20   0  305736   7404   6552 S   0.0   0.1   0:00.06 polkitd
   2194 root      10 -10   13852  13836  12004 S   0.0   0.1   0:00.00 iscsid
   2223 oracle-+  20   0 1241748  24116  13376 S   0.0   0.2   0:00.32 updater
   2235 root      20   0  163276   7392   5908 S   0.0   0.1   0:00.11 rsyslogd
   2241 root      20   0    8760   6524   5348 S   0.0   0.1   0:00.00 sshd
   2250 root      20   0  535832  45340  13392 S   0.0   0.5   0:00.68 fluentd
   2255 oracle-+  20   0 1241480  25716  15412 S   0.0   0.3   0:00.35 agent
   2300 chrony    20   0   10400   2984   2392 S   0.0   0.0   0:00.00 chronyd
   2351 root      20   0    8296   3472   2596 S   0.0   0.0   0:00.00 crond
   2353 root      20   0    4980   2344   2108 S   0.0   0.0   0:00.00 agetty
   2355 root      20   0    7548   2616   2360 S   0.0   0.0   0:00.00 agetty
   2365 oracle-+  20   0 1241292  25904  15344 S   0.0   0.3   0:00.31 gomon
   2415 root      20   0   18544   7936   6460 S   0.0   0.1   0:00.01 sudo
   2445 ocarun    20   0   22240  13464  10324 S   0.0   0.1   0:00.11 systemd
   2456 ocarun    20   0   14432   2904   1548 S   0.0   0.0   0:00.00 (sd-pam)
   2530 ocarun    20   0 1245024  28932  18520 S   0.0   0.3   0:00.18 runcomm+
   2537 pcp       20   0   14012   7688   5820 S   0.0   0.1   0:00.17 pmcd
   2551 root      20   0   13900   5096   4132 S   0.0   0.1   0:00.00 pmdaroot
   2566 root      20   0   14308   6448   4420 S   0.0   0.1   0:00.19 pmdaproc
   2579 root      20   0   13116   4828   3908 S   0.0   0.0   0:00.00 pmdaxfs
   2580 root      20   0   14032   6348   4668 S   0.0   0.1   0:00.15 pmdalin+
   2581 oracle-+  20   0 1241956  26044  15824 S   0.0   0.3   0:00.60 unified+
   2585 root      20   0   31304  23008  11324 S   0.0   0.2   0:00.05 python3
   2639 root      20   0   13096   4812   3896 S   0.0   0.0   0:00.00 pmdakvm
   2642 root      20   0   14636   5584   4544 S   0.0   0.1   0:00.00 pmdadm
   2647 pcp       20   0  262712  33464  13272 S   0.0   0.3   0:00.14 python3
   3356 pcp       20   0   15600   8292   5300 S   0.0   0.1   0:00.13 pmlogger
   3365 pcp       20   0   13916   6584   5328 S   0.0   0.1   0:00.00 pmie
   3381 pcp       20   0   12808   4160   3304 S   0.0   0.0   0:00.00 pmpause
   3493 pcp       20   0   12808   4168   3308 S   0.0   0.0   0:00.00 pmpause
   3957 root      20   0  526092  39020  13264 S   0.0   0.4   0:00.48 ruby
  10377 root      20   0   15496  10656   8632 S   0.0   0.1   0:00.00 sshd-se+
  10382 opc       20   0   22248  13480  10352 S   0.0   0.1   0:00.06 systemd
  10384 root      20   0   16612   7336   6260 S   0.0   0.1   0:00.00 systemd+
  10385 opc       20   0   14432   2912   1560 S   0.0   0.0   0:00.00 (sd-pam)
  10394 opc       20   0   15884   8120   5800 S   0.0   0.1   0:00.57 sshd-se+
  10395 opc       20   0    7212   6176   5496 S   0.0   0.1   0:00.00 bash
  10425 root      20   0   18552   9944   8460 S   0.0   0.1   0:00.03 sudo
  10427 root      20   0   16612   7464   6384 S   0.0   0.1   0:00.00 systemd+
  10428 root      20   0   18552   2832   1352 S   0.0   0.0   0:00.00 sudo
  10429 root      20   0   10088   7140   6324 S   0.0   0.1   0:00.00 su
  10430 root      20   0    7328   6220   5508 S   0.0   0.1   0:00.03 bash
  10512 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker+
  10721 root      20   0   15628  10648   8624 S   0.0   0.1   0:00.01 sshd-se+
  10724 root      20   0   16448   6920   5888 S   0.0   0.1   0:00.00 systemd+
  10725 opc       20   0   15888   8120   5800 S   0.0   0.1   0:00.00 sshd-se+
  10726 opc       20   0    7212   6156   5476 S   0.0   0.1   0:00.00 bash
  10756 opc       20   0   18036   3768   3048 S   0.0   0.0   0:07.49 iperf3
  10784 root      20   0   31896  26376   9788 S   0.0   0.3   0:00.10 python
  10799 root      20   0   10172   7220   5124 R   0.0   0.1   0:00.00 top
 1:  instance-20260614-1905.nabhaaspublicsu.nabhaasvcn.oraclevcn.com (10.10.3.88)   0.036ms reached
     Resume: pmtu 65535 hops 1 back 1 
[root@instance-20260614-1905 perf-results-client]#

Series Week 22 / 52 — OCI ExaCS and DB System Long Term Backups - Do they meet your compliance needs ?

Abhilash Kumar | Oracle ACE ♠ — Mon, 04 May 2026 19:40:31 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

Once you have moved to OCI Cloud - have you cross verified with Infformation Security Teams that the Company wide backup policies are in line with the backups with OCI Backups.

OCI Cloud backups are cloud defined but each oragnization policy has a different backup policy. A few things that information security team that regularly come up with are below

Regulatory Alignment
RPO / RTO
Custom Encryption Requirements
Retention Lifecycles
Geographic Sovereignty
Ransomware Defense

1. Ground Zero: Where Challenges Start

One main issue with Information Security Teams requirements to Databases in enterprises is understanding what does OCI provide out of the box

Relying solely on default OCI provisioning without an overlaid corporate strategy creates several operational and compliance vulnerabilities.

----------------------------------------------------------------------------------------------------
## 1. Ground Zero: Where Challenges Start
----------------------------------------------------------------------------------------------------

- "At a minimum" backups are created with the default Object Storage Bakcups provisioning available in OCI ExaCS and DB Systems

- There would be no Long Term Backup Strategy set up

The above two are typically the most common use cases OCI customers face.

2. Underneath Ground Zero:

Understanding Oracle Database Backups provide out of the box

Oracle OCI and DB System Backups provide two kinds of backups

OCI Object Storage Backups This is the default backup mecahnism for any OCI Class of databases Documentation here : OCI Backup Services

OCI Object Storage Backups does not have a retention mechanism for Long Term backups but it is possibile to have a standalone backup which does not expire , this is a crucial area where compliance misses are seen. I had this clarification by spending a lot of time on documentation and working with Oracle Support.

Autonomous Recovery Service Documentation here : ARS Documentation

----------------------------------------------------------------------------------------------------
## 2. Underneath Ground Zero: Finding the Real Problem

- One of teh difficult pills to swallow is that you have big gap of what is currently set up and what is needed for compliance

- Understand compliance/policy needs of the organization like Weekly, monthly , yearly backups and retention times 

- Identify and separate the clss of databases ( as ExaCS have a different mechanism to DB Systems ) 

- Once you have put them together you need to set upa. framework for long term backups 

- Remember a Backup is only as good as you can recover from it , so include sufficient testing plans 

----------------------------------------------------------------------------------------------------

3. Working Upwards:

There are multiple ways of setting up Long Term Backups , but what works best is how we have a framework called TAB ( Total Automation Box ) and how systematically we plug in organizational requirements into this framework

----------------------------------------------------------------------------------------------------
## 3. Working Upwards: From Understanding to Solution

- Segregate the requirements into technical terms 

- Identify the number of databases and the testing strategy for them 

- Identify a method to automate the Lomg Term backups in a reliable way and showcase to auditirs on demand 

----------------------------------------------------------------------------------------------------

How Nabhaas helps you

If you’ve made it this far, you already sense there’s a better way — in fact, you have a way ahead.

If you’d like Nabhaas to assist in your journey, remember — TAB is just one piece. Our Managed Delivery Service ensures your Oracle operations run smoothly between patch cycles, maintaining predictability and control across your environments.

TAB - Whitepaper ,
download here

Managed Delivery Services - Whitepaper ,
[download here](https://www.nabhaas.com/_files/ugd/dab815_96198a0627d64f75a3d3a2dce9bf185d.p

Series Week 21 / 52 — OCI Rolling Patches in Managed Services using TAB

Abhilash Kumar | Oracle ACE ♠ — Wed, 22 Apr 2026 06:54:18 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

In the world of Managed Services, patching is often treated as a chore. At Nabhaas, we treat it as a science. To achieve true operational excellence, organizations must transition through three distinct phases of maturity. Here is how we use our Total Automation Box (TAB) to move from infrastructure chaos to business agility:

Patching does not always need a downtime especially in High Availability always on environments , Oracle provides the facility of rolling apply , it it the art of support teams to plug the patching windows with minimal impact.

1. Ground Zero: Where Challenges Start

----------------------------------------------------------------------------------------------------
## 1. Ground Zero: Where Challenges Start

- The Reality of Legacy Systems: In many environments, patching is still treated as a manual, reactive, and high-risk operation.

- The Downtime Dilemma: Fear of application downtime heavily influences maintenance schedules, frequently causing critical patches to be pushed back or delayed.

- The High Availability Exception: Not every class of database requires an downtime. 

- In High Availability (HA) environments, such as Oracle Cloud Infrastructure (OCI) ExaCS, zero-downtime rolling patches are highly applicable and should be leveraged to avoid disruption.


----------------------------------------------------------------------------------------------------

2. Underneath Ground Zero:

----------------------------------------------------------------------------------------------------
## 2. Underneath Ground Zero: Finding the Real Problem

- Strategic Architecture Review: The essential foundational step is to strictly segregate databases into two distinct categories: High Availability (HA) and Non-High Availability.

- Window Categorization: This architectural division automatically sorts your patching strategy, establishing clear boundaries between rolling (zero-downtime) and non-rolling (downtime-required) patching windows.


----------------------------------------------------------------------------------------------------

3. Working Upwards:

----------------------------------------------------------------------------------------------------
## 3. Working Upwards: From Understanding to Solution

- Targeted Playbooks: To move from theory to execution, prepare separate, dedicated patching playbooks tailored specifically for HA environments versus non-HA environments.

- Operational Predictability: Creating these specialized playbooks transforms complex, high-stress maintenance into structured, highly implementable patching windows.

----------------------------------------------------------------------------------------------------

How Nabhaas helps you

If you’ve made it this far, you already sense there’s a better way — in fact, you have a way ahead.

TAB - Whitepaper ,
download here

Managed Delivery Services - Whitepaper ,
download here

The Oracle MOS Shortcut: A Life-Saver for P1 Issues

Abhilash Kumar | Oracle ACE ♠ — Mon, 20 Apr 2026 12:14:50 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

If you’ve ever worked on a Priority 1 (P1) "Production Down" event ( which most of us DBA's would have) , you know that every second counts. Navigating through the My Oracle Support (MOS) dashboard can feel like running through sand when you're stressed—the widgets take time to load, the tabs can be laggy, and searching for an SR ID manually is a click-heavy chore. There is a much better way.

By using a direct-link utility, you can jump straight to your Service Request (SR) from your browser address bar or a chat window.

The "Direct-Link" and it is super simple

MOS SR works on a X-XXXXXXXXXX ( single_digit-hyphen_ten_digit )

To bypass the standard navigation and go straight to the SR details, use this URL structure:


https://support.oracle.com/ic/builder/rt/customer_portal/live/webApps/customer-portal/?SR=X-XXXXXXXXXX

PS : I found this accidentally while working on numerous SR's , this works well for Open and closed SR's

How to use it:
1. Copy the link.
2. Replace X-XXXXXXXXXX with your actual SR number (e.g., 3-3141592653).
3. Paste & Go. You’ll be dropped directly into the Customer Portal view of that specific SR.

A Life-Saver for P1 Issues

Zero Dashboard Latency
The standard MOS homepage pulls a lot of data (news, health checks, multiple SR widgets). In a crisis, you don't need the noise. This link targets the Customer Portal builder directly, which is a lightweight interface that loads significantly faster.
War Room Coordination
When you’re on a bridge call or in a Slack/Teams war room, don't just say "check the update on the SR." Drop this link. It ensures every stakeholder—from the DBA to the IT Director—is looking at the exact same update instantly.
Effortless Re-Authentication

Oracle MOS sessions eventually timeout. Usually, this means being kicked back to the login screen and then dumped on the homepage, forcing you to find your SR all over again. If your session expires, clicking this direct link will take you to the SSO login page, and once you authenticate, it will redirect you straight back to the SR. No lost context.

Ease Your Life with "Auto-Refresh" Tabs

During a P1, you are often waiting with bated breath for an update from an Oracle engineer. Manually hitting F5 every 30 seconds is a distraction you don't need.
The Strategy:

Open your SR using the direct link.
Use a Browser Refresh Extension that suits your browser
Set a tab to auto-refresh every 2 to 5 minutes.

By keeping this "Live" tab on a second monitor, you can focus on logs or troubleshooting, knowing that the moment the Oracle engineer posts an update, it will appear on your screen without you lifting a finger.

Summary

In the world of Oracle Support, efficiency is king. Bookmark this URL structure or set up a browser keyword (like mos) to transform your P1 response from "searching and clicking" to "acting and resolving."

Series Week 20 / 52 — Differentiating Patching differences between Exadata On Prem and OCI Databases (DB Systems and ExaCS)

Abhilash Kumar | Oracle ACE ♠ — Sun, 05 Apr 2026 06:45:44 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

Moving on from the previous week 19 ( from my previous blog )

https://dev.to/nabhaas/week-19-52-oracle-databases-choosing-between-on-prem-or-any-cloud-oci-aws-gcp-azure--5b0p

While understanding the ExaCS vs Exadata On Prem , we are trying to comprehend the patching differences to manage Exadata On Prem vs ExaCS , a cloud environment is still a war when compared to On Prem nevertheless its still the DBA's capability the rules to manage these machines.

Some useful refresher for Oracle On Prem - Exadata Patching

https://blogs.oracle.com/exadata/exadata-mos-notes
https://www.oracle.com/a/tech/docs/exadata-software-maintenance-2022.pdf

Some useful refresher for Oracle OCI - ExaCS Patching

https://docs.oracle.com/en-us/iaas/exadatacloud/doc/ecs-patch-update.html

A quick tabular differences of On Prem Patching and OCI Class of databases


-- On Prem
+----------------------+---------------------------+
| Component            | On-Prem Exadata           | 
+----------------------+---------------------------+-
| Cell Patching        | Quarterly (Manual)        | 
| Infiniband Patching  | Quarterly (Manual)        | 
| Dom0 Patching        | Not Applicable            | 
| DomU VM Patching     | Not Applicable            | 
| Grid Infrastructure  | Quarterly (Manual)        | 
| Database Patching    | Quarterly (Manual)        | 
+----------------------+---------------------------+


-- OCI ExaCS 
+----------------------+---------------------------+----------------------------------------------------------------+
| Component            | On-Prem Exadata           | ExaCS (OCI)                                                    |
+----------------------+---------------------------+----------------------------------------------------------------+
| Cell Patching        | Quarterly (Manual)        | Managed by Oracle (Implicit) as a part of Dom0 auto schedules  |
| Infiniband Patching  | Quarterly (Manual)        | Managed by Oracle (Implicit) as a part of Dom0 auto schedules  |
| OS Patching          | DB Node Patching (Manual) | Quarterly (Customer Driven)                                    |
| Grid Infrastructure  | Quarterly (Manual)        | Quarterly (Customer Driven)                                    |
| Database Patching    | Quarterly (Manual)        | Quarterly (Customer Driven)                                    |
+----------------------+---------------------------+----------------------------------------------------------------+


-- OCI DB Systems 
+----------------------+---------------------------+----------------------------------------------------------------+
| Component            | On-Prem Exadata           | DB System (OCI)                                                |
+----------------------+---------------------------+----------------------------------------------------------------+
| Cell Patching        | Not Applicable            | Not Applicable                                                 |
| Infiniband Patching  | Not Applicable            | Not Applicable                                                 |
| OS Patching          | OS specific               | Bundled with Grid Patcing                                      |
| Grid Infrastructure  | Quarterly (Manual)        | Quarterly (Customer Driven)                                    |
| Database Patching    | Quarterly (Manual)        | Quarterly (Customer Driven)                                    |
+----------------------+---------------------------+----------------------------------------------------------------+

1. Ground Zero: Where Challenges Start

----------------------------------------------------------------------------------------------------
## 1. Ground Zero: Where Challenges Start - Understand Patching Problems
----------------------------------------------------------------------------------------------------


- Massive Infrastructure Footprint: You are managing multiple ExaCS VM Clusters scattered 
across different OCI compartments, each with its own lifecycle and governance needs.

- Application Silos: Your databases are logically segregated to support 
diverse application stacks. This isn't just one database; it’s an entire fleet 
of distinct environments.

- The Scale Paradox: With dozens of databases across multiple clusters, the sheer volume 
of "moving parts" makes manual oversight impossible. You aren't just a DBA anymore; 
you are a Fleet Manager.

2. Underneath Ground Zero:

----------------------------------------------------------------------------------------------------
## 2. Underneath Ground Zero: Finding the Real Problem ( problem of plenty )
----------------------------------------------------------------------------------------------------


- The Maintenance Collision: Between Dom0 (Hypervisor), DomU (Guest VM), 
Grid Infrastructure, and the Database layer, the sheer number of required updates 
(for both Prod and Non-Prod) far exceeds the number of available maintenance windows.

- The Downtime Crunch: Business units are demanding 24/7 availability. 
Finding a slot to take down a cluster for a rolling patch is a high-stakes 
negotiation with stakeholders.

- The Certification Bottleneck: Applications are stubborn. Some require specific 
DB versions (e.g., 19.18 vs 19.21) for certification. 
You can’t just "patch everything to latest"  without breaking the application tier.

- Version Fragmentation: You need the surgical ability to maintain multiple database 
versions on the same infrastructure to satisfy different application compatibility 
requirements without compromising the security of the whole cluster.

3. Working Upwards:


----------------------------------------------------------------------------------------------------
## 3. Working Upwards: From Understanding to Solution - Simplify and Automate
----------------------------------------------------------------------------------------------------

From the above it is evident you need the following 

- Patching automation 
- Multiple and Flexbile patching windows 
- Logical Patching windows ( e,g. patch all GI in a window and DB in another window )
- Logical Patching methods ( patch Dom-U VM , GI , DB ) in that order for every cluster  

Now you get to see the patching conundrum 

TAB from Nabhaas does this for you 

- The Orchestration Engine: TAB (Total Automation Box) isn't just a process; 
it’s an automation framework designed to handle the "heavy lifting" of 
ExaCS Databases lifecycle management.

- Strategic Pre-checks: TAB runs exhaustive, automated health checks before the first 
line of code is patched. If the cluster isn’t healthy, TAB stops the process before the 
downtime starts.

- Parallel Fleet Patching: While manual patching is linear (one by one), 
TAB allows for parallelized, orchestrated updates across multiple VM clusters, 
drastically reducing the total "time-to-compliance."

- Version Pinning & Flexibility: TAB gives the CTO the power to patch 
the Grid Infrastructure  to the latest secure version while "pinning" specific 
Databases to the exact versions required by application owners.

- Predictable Rollbacks: In the rare event of a patch failure, TAB provides a structured, 
rollback path, ensuring that "failed patching" doesn't turn into "extended outage."

- Governance at Scale: TAB provides a single pane of glass for patching status across all 
compartments, transforming a chaotic spreadsheet exercise into a predictable, 
audit-ready dashboard.

Eventually you can see that OCI Patching is an orbital shift to the traditional way of Exadata Patching.

How Nabhaas helps you

If you’ve made it this far, you already sense there’s a better way — in fact, you have a way ahead.

TAB - Whitepaper ,
download here

Managed Delivery Services - Whitepaper ,
download here

Series Week 19 / 52 — Oracle Databases choosing between On Prem or any Cloud [ OCI , AWS , GCP , Azure ]

Abhilash Kumar | Oracle ACE ♠ — Fri, 27 Mar 2026 05:08:40 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

Oracle database can be hosted in all major cloud vendors [ OCI , AWS , GCP , Azure ] , there is a lot of reference material here.
https://www.oracle.com/in/cloud/multicloud/

This blog is about going through the decision making process of a CTO which cloud or hybrid setup to use

Organisations already have Exadata Infra and are looking to upgrade their infra
Organizations that are trying to implement new Exadata and are considering cloud solutions
Organizations that are into AWS, Azure , GCP trying to make best use of Oracle databases

1. Ground Zero: Where Challenges Start


> Organizations already running Exadata (On-Prem)
  Many are approaching a hardware refresh cycle. While the system still works,
  questions around cost, utilization, and long-term sustainability begin to surface.
  - The pressure to “move to cloud” often starts here — sometimes without full clarity.
  - There are regulatrory requirements to stay On Prem


> Organizations planning new Exadata adoption
  These teams are at the beginning of their journey, trying to balance a large upfront
  investment against cloud-based flexibility. In most cases, workload sizing is still
  unclear, making decisions heavily influenced by vendor narratives.

> Organizations already in AWS / Azure / GCP
  Here, Oracle databases are often running on non-native platforms, leading to
  compromises in performance, licensing, or scalability. OCI enters the conversation
  as a specialized option — but only for the database layer.

> Organizations evaluating Autonomous vs ExaCS
  There is growing curiosity around Autonomous Database, often seen as the “future.”
  At the same time, there is hesitation — especially where applications require control,
  customization, or certified environments.

>> Different starting points, but all leading to the same question:
   What is the right platform and service model for our databases?

2. Underneath Ground Zero: Finding the Real Problem

> Existing Exadata Customers
  The decision is often framed too narrowly — as a choice between upgrading hardware
  or moving to the cloud. What is usually missing is a deeper look at how the system
  is actually being used, and what it truly costs to operate.

> New Exadata Adopters
  Without clear workload insights, there is a tendency to overestimate requirements.
  This leads to over-sizing, higher costs, and unnecessary complexity from day one.

> Multi-Cloud Organizations
  OCI is sometimes treated as just another cloud provider, rather than a platform
  optimized specifically for Oracle databases. This leads to gaps in network design,
  identity integration, and overall architecture.
  Organizations use heavy applications native to AWS , Azure , GCP , 
  the egress cost would be a silent killer with multi cloud [ unless you have a dedicated bandwidth provider ]

> Autonomous vs ExaCS Decision Gap
  Autonomous is often assumed to be the default choice, but not all applications fit.
  Enterprise systems like EBS or heavily customized databases may require capabilities
  that Autonomous intentionally abstracts away.

> The Common Thread
  Across all these scenarios, decisions are driven more by trends and assumptions
  than by actual workload behavior and application needs.

>> The real challenge is understanding the service model behind it.

3. Working Upwards: From Understanding to Solution


> For Existing Exadata Customers
  If workloads are stable and predictable, continuing on-prem with an optimized
  refresh strategy can still make sense. However, if demand fluctuates or data center
  overhead becomes a concern, ExaCS offers a more flexible path forward.

> For New Exadata Adopters
  The focus should be on understanding the workload first — usage patterns, peak
  demand, and I/O behavior. Starting small and scaling gradually is often more
  effective than committing to a large footprint upfront.

> For Multi-Cloud Organizations
  OCI should be positioned thoughtfully — not as a generic cloud, but as a
  database-focused platform. Network design, identity alignment, and cost
  considerations need to be handled deliberately.

> Autonomous vs ExaCS — Making the Right Call

  Autonomous Database works best when:
    - Workloads are standardized
    - Operational overhead needs to be minimized
    - Full automation is acceptable

  Autonomous Database does not work best when:

    - Application stack version compatibility needs to be done regularly due to complex app stack.


  ExaCS is more suitable when:
    - Fine-grained control over the database is required
    - Applications have specific certification or dependency needs
    - Custom configurations and integrations are critical

Bringing It All Together

The most effective decisions come from aligning three things:
- The nature of the workload
- The level of control required
- The operational model the organization is ready for

CTO Outcome : Instead of forcing a single direction, the goal is to place each workload on the platform where it performs best — with clarity, control, and predictability.

The right choice depends on the application and the service model , the challenge is to understand the goal and identify the means.

How Nabhaas helps you

If you’ve made it this far, you already sense there’s a better way — in fact, you have a way ahead.

TAB - Whitepaper ,
download here

Managed Delivery Services - Whitepaper ,
download here

Series Week 18 / 52 — Managed Cost Optimization: Controlling OCI Database Spends

Abhilash Kumar | Oracle ACE ♠ — Sun, 08 Mar 2026 16:30:20 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

Moving on to a common topic of costs , an oversight on operations can dent a hole in Cloud databse spends.

The promise of the cloud—predictable costs and seamless elasticity—often clashes with the reality of OCI-hosted Oracle environments. When costs spiral, the culprit is rarely the platform itself; rather, it's an architecture that lacks built-in cost governance.

Expensive inefficiencies usually stem from three common oversights:

Static Provisioning: Databases are sized for "worst-case" peak loads and left that way indefinitely.

Idle Resources: Non-production environments remain active 24/7, burning budget while teams are offline.

Data Bloat: Backups, archivelogs, and stagnant data accumulate silently without a lifecycle review.

True cost optimization in OCI isn't a post-mortem finance meeting; it is a continuous operational discipline. By shifting from passive consumption to deliberate management, a CTO can transform OCI from a source of budget surprises into one of the most transparent and controllable assets in the enterprise portfolio.

1. Ground Zero: Where Challenges Start

+--------------------------------------------------------------------------------------------------+
| 1. Ground Zero: Where Challenges Start                                                           |
+--------------------------------------------------------------------------------------------------+
| - Databases sized for peak load but running idle most of the time                                |
| - Non-production environments running 24x7                                                       |
| - Over-provisioned Exadata or DB Systems                                                         |
| - Storage growth not monitored strategically                                                     |
| - Multiple small databases instead of consolidation                                              |
| - Cloud costs reviewed only during budget cycles                                                 |
|                                                                                                  |
| Typical Symptoms                                                                                 |
| • Development environments cost nearly as much as production                                     |
| • Storage consumption grows without clear ownership                                              |
| • Compute capacity rarely adjusted after initial deployment                                      |
| • Test environments forgotten after project completion                                           |
|                                                                                                  |
| >> Cloud makes scaling easy.                                                                     |
|    But without governance, it also makes waste easy.                                             |
+--------------------------------------------------------------------------------------------------+

2. Underneath Ground Zero:



+--------------------------------------------------------------------------------------------------+
| 2. Underneath Ground Zero: Finding Where OCI Costs Can Be Controlled                             |
+--------------------------------------------------------------------------------------------------+
|                                                                                                  |
| Compute Optimization                                                                             |
| • Right-sizing OCPU allocations for DB Systems                                                   |
| • Scaling ExaCS compute nodes only when required                                                 |
| • Using burstable or smaller shapes for non-prod workloads                                       |
|                                                                                                  |
| Storage Optimization                                                                             |
| • Monitoring ASM diskgroup utilization                                                           |
| • Removing unused tablespaces and orphaned datafiles                                             |
| • Archivelog retention optimization                                                              |
|                                                                                                  |
| Environment Lifecycle Management                                                                 |
| • Scheduling non-prod shutdowns (nights/weekends)                                                |
| • Decommissioning unused environments                                                            |
| • Refreshing non-prod with smaller configurations                                                |
|                                                                                                  |
| Consolidation Opportunities                                                                      |
| • Multiple schemas instead of multiple databases                                                 |
| • PDB-based consolidation in multitenant environments                                            |
| • Shared infrastructure for low-volume workloads                                                 |
|                                                                                                  |
| Data Growth Control                                                                              |
| • Partition lifecycle management                                                                 |
| • Archiving historical data                                                                      |
| • Purging unused application logs                                                                |
|                                                                                                  |
| Licensing Awareness                                                                              |
| • Aligning OCPU counts with license entitlements                                                 |
| • Avoiding accidental over-allocation of licensed cores                                          |
|                                                                                                  |
| Backup and DR Strategy                                                                           |
| • RMAN retention policy tuning                                                                   |
| • Evaluating DR environment sizing                                                               |
| • Storage tier optimization for backups                                                          |
|                                                                                                  |
| Monitoring and Visibility                                                                        |
| • Cost tracking by environment and business unit                                                 |
| • Tagging resources for financial accountability                                                 |
|                                                                                                  |
| >> Cost optimization is rarely one big decision.                                                 |
|    It is dozens of small architectural choices.                                                  |
+--------------------------------------------------------------------------------------------------+

3. Working Upwards:

+--------------------------------------------------------------------------------------------------+
| 3. Working Upwards: From Understanding to Solution                                               |
+--------------------------------------------------------------------------------------------------+
| - Establish cost visibility dashboards for OCI database services                                 |
| - Align environment sizing with real workload patterns                                           |
| - Implement scheduled shutdown for non-prod environments                                         |
| - Consolidate low-utilization databases where appropriate                                        |
| - Introduce periodic storage and archivelog audits                                               |
| - Align DR architecture with realistic recovery objectives                                       |
| - Integrate cost review into operational governance                                              |
|                                                                                                  |
| Mature OCI Cost Governance Model                                                                 |
| • Quarterly architecture reviews                                                                 |
| • Automated non-prod lifecycle controls                                                          |
| • Storage growth alerts                                                                          |
| • Business-unit cost accountability                                                              |
|                                                                                                  |
| CTO Outcome                                                                                      |
| • Predictable cloud spending                                                                     |
| • Infrastructure aligned with business value                                                     |
| • Reduced waste without reducing capability                                                      |
|                                                                                                  |
| >> Cloud efficiency is not about spending less.                                                  |
|    It is about spending intentionally.                                                           |
+--------------------------------------------------------------------------------------------------+

How Nabhaas helps you

If you’ve made it this far, you already sense there’s a better way — in fact, you have a way ahead.

TAB - Whitepaper ,
download here

Managed Delivery Services - Whitepaper ,
download here

Series Week 17 / 52 — Benchmarking Oracle Service KPIs: Uptime, MTTR, MTBF

Abhilash Kumar | Oracle ACE ♠ — Fri, 27 Feb 2026 07:53:08 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

Moving on to the hot topic on on Clouds for enterprises - Uptime , MTTR and MTBF

Cloud providers advertise 99.99% or 99.999% availability, but very few teams pause to understand what is actually being promised. Availability SLAs are defined at the service infrastructure layer — not necessarily at the business service layer. The SLA may cover compute instance availability, storage durability, or control plane uptime — but it does not automatically include application responsiveness, database performance degradation, misconfiguration, architectural saturation, or customer-managed failover delays. The devil is in the definitions: what counts as downtime, what is excluded (planned maintenance, regional events, customer actions), and how availability is measured.

MTTR ( Mean Time to Repair ) - For OCI Databases ( DB Systems and ExaCS ) , this is customer defined For Autonomous Databases is this OCI defined

Therefor MTBF ( Mean Time Between Failures ) - is a maturity indicator.

ExaCS HA Documentation here

ADB HA Documentation here

1. Ground Zero: Where Challenges Start


+--------------------------------------------------------------------------------------------------+
| 1. Ground Zero: Where Challenges Start                                                           |
+--------------------------------------------------------------------------------------------------+
| - Uptime reported as “database was open”                                                         |
| - No distinction between planned vs unplanned downtime                                           |
| - MTTR calculated inconsistently                                                                 |
| - Incident severity not standardized                                                             |
| - No historical trend tracking                                                                   |
|                                                                                                  |
| Typical KPI Mistakes                                                                             |
| • Counting partial outages as “available”                                                        |
| • Ignoring performance degradation in uptime calculation                                         |
| • Measuring MTTR from ticket assignment, not incident start                                      |
| • No clear incident start/end timestamp discipline                                               |
| • DR failovers counted as uptime improvement without context                                     |
|                                                                                                  |
| >> Numbers exist. Reliability insight does not.                                                  |
+--------------------------------------------------------------------------------------------------+

2. Underneath Ground Zero:

+--------------------------------------------------------------------------------------------------+
| 2. Underneath Ground Zero: Finding the Real Problem                                              |
+--------------------------------------------------------------------------------------------------+
| - SLAs defined commercially, not operationally                                                   |
| - Monitoring tools not aligned with business impact                                              |
| - No separation between infrastructure and application outage                                    |
| - Root cause trends not tied to KPI analysis                                                     |
| - No MTBF tracking to identify systemic instability                                              |
|                                                                                                  |
| Hidden Structural Issues                                                                         |
| • Frequent small incidents masking instability                                                   |
| • Reactive firefighting improves MTTR but not MTBF                                               |
| • Patching windows inflating downtime metrics                                                    |
| • Global teams measuring differently                                                             |
| • Lack of service-level ownership                                                                |
|                                                                                                  |
| Core Reality                                                                                     |
| You cannot improve what you don’t measure correctly.                                             |
| And you cannot measure correctly without consistent definitions.                                 |
+--------------------------------------------------------------------------------------------------+

3. Working Upwards:

+--------------------------------------------------------------------------------------------------+
| 3. Working Upwards: From Understanding to Solution                                               |
+--------------------------------------------------------------------------------------------------+
| - Define precise KPI definitions                                                                 |
|   • Uptime = Service availability excluding approved maintenance                                 |
|   • MTTR = Time from incident detection to full service restoration                              |
|   • MTBF = Total uptime / number of unplanned failures                                           |
|                                                                                                  |
| - Standardize incident classification & severity models                                          |
| - Automate incident timestamp capture                                                            |
| - Separate performance degradation from full outages                                             |
| - Track rolling 30/90/180 day KPI trends                                                         |
| - Correlate KPIs with change events (patches, deployments, upgrades)                             |
| - Benchmark across environments (Prod vs DR vs Non-Prod)                                         |
|                                                                                                  |
| Mature Oracle Service KPI Model                                                                  |
| • Uptime tied to business service, not instance status                                           |
| • MTTR trending downward through structured response                                             |
| • MTBF increasing as architectural stability improves                                            |
| • KPI dashboards shared with leadership                                                          |
|                                                                                                  |
| CTO Outcome                                                                                      |
| • Data-driven reliability decisions                                                              |
| • Justified architecture investments                                                             |
| • Predictable SLA adherence                                                                      |
| • Fewer executive surprises                                                                      |
|                                                                                                  |
| >> Uptime is a result.                                                                           |
|    MTTR is a capability.                                                                         |
|    MTBF is a maturity indicator.                                                                 | 
+--------------------------------------------------------------------------------------------------+

How Nabhaas helps you

If you’ve made it this far, you already sense there’s a better way — in fact, you have a way ahead.

TAB - Whitepaper ,
download here

Managed Delivery Services - Whitepaper ,
download here

Series Week 16 / 52 — Automating Prechecks & Postchecks: Managed Services vs DIY

Abhilash Kumar | Oracle ACE ♠ — Mon, 16 Feb 2026 10:04:03 +0000

{ Abhilash Kumar Bhattaram : Follow on LinkedIn }

We have talked about how to automated OCI Patching in our Week 5 Blog (here) , lets now talk about automating Prechecks / Post checks , not all of them can be automated but significant portion of it can be done if attention is paid to Multiple use cases of Prechecks / Post checks

In a surgery, a doctor doesn’t walk into the OT and “figure it out.” Vitals are checked. Instruments are counted. Anesthesia readiness is confirmed. Post-procedure monitoring is predefined.

That level of discipline is expected in medicine. It should be expected in database operations too.

Prechecks and postchecks are are what separate controlled change from accidental impact. Not everything can be automated —
but a significant portion can, if designed intentionally.

1. Ground Zero: Where Challenges Start


+--------------------------------------------------------------------------------------------------+
| 1. Ground Zero: Where Challenges Start                                                          |
+--------------------------------------------------------------------------------------------------+
| - Prechecks live in emails, Word documents, or tribal knowledge                                 |
| - Postchecks depend on individual DBA experience                                                |
| - No standardized validation across environments                                                |
| - Evidence not captured consistently for audits                                                 |
| - Change windows consumed by manual verification                                                |
|                                                                                                 |
| Common Use Cases for Prechecks / Postchecks                                                     |
|                                                                                                 |
| • Patching Activities                                                                           |
|   - RMAN backup validation                                                                      |
|   - Invalid object count comparison                                                             |
|   - Tablespace & diskgroup free space                                                           |
|   - Cluster / CRS health                                                                        |
|   - AWR snapshot before & after                                                                 |
|                                                                                                 |
| • Deployment Activities                                                                         |
|   - Schema version validation                                                                   |
|   - Dependency verification                                                                     |
|   - Rollback script availability                                                                |
|   - Session spike monitoring post deployment                                                    |
|                                                                                                 |
| • Maintenance Activities                                                                        |
|   - DR sync status                                                                              |
|   - Archive log growth monitoring                                                               |
|   - ASM rebalance status                                                                        |
|   - Blocking session check                                                                      |
|                                                                                                 |
| • Upgrade / Migration Activities                                                                |
|   - COMPATIBLE parameter review                                                                 |
|   - Deprecated parameter scan                                                                   |
|   - Feature usage analysis                                                                      |
|                                                                                                 |
| • Security & Compliance Pre/Post Checks                                                         |
|   - Account lockout & password complexity profile validation                                    |
|   - Excessive privilege grants (DBA/SYS role review)                                            |
|   - Sensitive table / column access audit review                                                |
|   - Oracle Unified Auditing enabled & policy validation                                         |
|   - Audit retention policy enforcement                                                          |
|   - TDE key rotation status verification                                                        |
|   - Database Vault realm & rule validation                                                      |
|   - Fine-Grained Auditing (FGA) policy verification                                             |
|                                                                                                 |
| • Organizational / Process Validation                                                           |
|   - Change approval confirmation                                                                |
|   - Business window validation                                                                  |
|   - Stakeholder communication confirmation                                                      |
|                                                                                                 |
| >> Managed services often execute changes.                                                      |
|    Mature services validate readiness before and after execution.                               |
+-------------------------------------------------------------------------------------------------+

2. Underneath Ground Zero:


+--------------------------------------------------------------------------------------------------+
| 2. Underneath Ground Zero: Finding the Real Problem                                              |
+--------------------------------------------------------------------------------------------------+
| - Knowledge lives in people, not systems                                                         |
| - Checklists are not version-controlled                                                          |
| - No measurable readiness score                                                                  |
| - Validation steps skipped during “urgent” changes                                               |
| - DIY scripts automate execution — but not validation                                            |  
|                                                                                                  |
| Hidden Risks                                                                                     |
| - Patch applied on unstable baseline                                                             |
| - Deployment approved without rollback validation                                                |
| - DR drift unnoticed                                                                             |
| - Security misconfigurations discovered during audits                                            |
| - RCA becomes subjective (“we thought it was fine”)                                              |
|                                                                                                  |
| Core Issue                                                                                       |
| Execution gets automated.                                                                        |
| Validation does not.                                                                             |
|                                                                                                  |
| And without structured validation, predictability becomes fragile.                               |
+--------------------------------------------------------------------------------------------------+

3. Working Upwards:


+--------------------------------------------------------------------------------------------------+
| 3. Working Upwards: From Understanding to Solution                                               |
+--------------------------------------------------------------------------------------------------+
| - Convert recurring manual checks into structured automation modules                             |
| - Standardize precheck/postcheck templates across environments                                   |
| - Integrate validation into patching, deployment, and maintenance workflows                      |
| - Capture before/after state snapshots automatically                                             |
| - Produce readiness scorecards for every change                                                  |
| - Maintain auditable evidence logs                                                               |
| - Keep expert judgment manual — automate repeatable validation                                   |
|                                                                                                  |
| Managed Services Advantage                                                                       |
| • Institutional memory embedded into process                                                     |
| • Consistent validation across teams                                                             |
| • Faster, safer execution windows                                                                |
| • Audit-ready change documentation                                                               |
|                                                                                                  |
| DIY Limitation                                                                                   |
| • Automation focuses on “doing”                                                                  |
| • Mature services focus on “verifying”                                                           |
|                                                                                                  |
| >> In surgery, the checklist prevents avoidable mistakes.                                        |
|    In database operations, automated prechecks and postchecks prevent avoidable outages.         |
+--------------------------------------------------------------------------------------------------+

How Nabhaas helps you

If you’ve made it this far, you already sense there’s a better way — in fact, you have a way ahead.

TAB - Whitepaper ,
download here

Managed Delivery Services - Whitepaper ,
download here