DEV Community

Vahid Yousefzadeh
Vahid Yousefzadeh

Posted on

PDB Recovery Isolation

#devops #oracle #database #linux

In Oracle Database version 19c, Data Guard cannot manage operations such as hot cloning or point-in-time recovery (PITR) at the PDB level. If such operations are performed on the primary database, Data Guard will ignore that PDB without exiting the recovery mode and continue applying redo for the remaining PDBs.

–Data Guard 19c

SQL> select OPEN_MODE from v$database;
OPEN_MODE
——————–
READ ONLY WITH APPLY

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

———- —————————— ———- ———-
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ ONLY  NO
Enter fullscreen mode Exit fullscreen mode 
–Primary 19c

SQL> create pluggable database pdb2  from pdb1;
Pluggable database created.

SQL> alter pluggable database pdb2 open;
Pluggable database altered.

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
———- —————————— ———- ———-
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
         5 PDB2                           READ WRITE NO
Enter fullscreen mode Exit fullscreen mode 
–Data Guard 19c

SQL> select OPEN_MODE from v$database;
OPEN_MODE
——————–
READ ONLY WITH APPLY

SQL> alter pluggable database pdb2 open;
ORA-01111: name for data file 13 is unknown – rename to correct file

SQL> select name,status from v$datafile;
NAME                                STATUS
———————————– ——-
/oracle19c/home/dbs/UNNAMED00013    SYSOFF
/oracle19c/home/dbs/UNNAMED00014    RECOVER
/oracle19c/home/dbs/UNNAMED00015    RECOVER
/oracle19c/home/dbs/UNNAMED00016    RECOVER
Enter fullscreen mode Exit fullscreen mode

Data files on the standby remain in SYSOFF or RECOVER status, showing that the new PDB is not properly synchronized.

In Oracle 21c, the PDB Recovery Isolation feature was introduced to solve this issue. With this enhancement, a background process is spawned that restores and recovers the newly cloned PDB (or the PDB that has undergone point-in-time recovery).

During this process, the CDB never leaves recovery mode, and for a short time, two types of Managed Recovery Processes (MRP) coexist:

  • One for the CDB-level recovery
  • Another for the specific PDB recovery

Once the PDB is fully synchronized with the CDB, the PDB-level MRP automatically stops, and the two MRPs merge into a single recovery stream.

– Data Guard 21c

SQL> select OPEN_MODE from v$database;
OPEN_MODE
——————–
READ ONLY WITH APPLY

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
———- —————————— ———- ———-
         2 PDB$SEED                       READ ONLY  NO
         5 PDB1                           READ ONLY  NO
Enter fullscreen mode Exit fullscreen mode 
–Primary 21c

SQL> create pluggable database pdb2  from pdb1;
Pluggable database created.

SQL> alter pluggable database pdb2 open;
Pluggable database altered.

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
———- —————————— ———- ———-
         2 PDB$SEED                       READ ONLY  NO
         3 PDB2                           MOUNTED
         5 PDB1                           READ ONLY  NO

SQL> alter pluggable database pdb2 open;
Pluggable database altered.

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
———- —————————— ———- ———-
         2 PDB$SEED                       READ ONLY  NO
         4 PDB2                           READ WRITE NO
         5 PDB1                           READ WRITE NO
Enter fullscreen mode Exit fullscreen mode 
–Data Guard 21c

SQL> select name,CON_ID,RECOVERY_STATUS from v$pdbs where NAME=’PDB2′;
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB2                4 DISABLED AUTOMATIC RECOVER

SQL> !sleep 60

SQL> /
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB2                4 ENABLED

SQL> alter pluggable database pdb2 open;
Pluggable database altered.

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
———- —————————— ———- ———-
         2 PDB$SEED                       READ ONLY  NO
         4 PDB2                           READ ONLY  NO
         5 PDB1                           READ ONLY  NO
Enter fullscreen mode Exit fullscreen mode

Now PDB2 has been successfully restored and recovered on the standby.
A look at the alert log confirms the automated recovery steps and MRP handoff process.

Challenges

Challenge 1: Standby in MOUNT Mode
If the Data Guard standby is in MOUNT mode, the PDB Recovery Isolation process waits until the standby is opened. Once opened, it immediately starts the recovery operation for the affected PDB.

–Data Guard

SQL> startup mount force;

SQL> alter database recover managed standby database disconnect;
Enter fullscreen mode Exit fullscreen mode 
–Primary

SQL> create pluggable database pdb3  from pdb1;
Pluggable database created.

SQL> alter pluggable database pdb3 open;
Pluggable database altered.
Enter fullscreen mode Exit fullscreen mode 
–Data Guard

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
———- —————————— ———- ———-
         2 PDB$SEED                       MOUNTED
         3 PDB3                           MOUNTED
         4 PDB2                           MOUNTED
         5 PDB1                           MOUNTED


SQL> select name,CON_ID,RECOVERY_STATUS from v$pdbs;
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB$SEED        2 ENABLED
PDB3            3 DISABLED AUTOMATIC RECOVER
PDB2            4 ENABLED
PDB1            5 ENABLED
Enter fullscreen mode Exit fullscreen mode

After the standby is opened, AUTOMATIC RECOVER for PDB3 starts automatically, and within a short period, its RECOVERY_STATUS changes to ENABLED:

–Data Guard

SQL> alter database recover managed standby database cancel;
Database altered.

SQL> alter database open;
Database altered.

SQL> alter database recover managed standby database disconnect;

SQL> select name,CON_ID,RECOVERY_STATUS from v$pdbs;
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB$SEED            2 ENABLED
PDB3                3 DISABLED AUTOMATIC RECOVER
PDB2                4 ENABLED
PDB1                5 ENABLED

SQL> /
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB$SEED            2 ENABLED
PDB3                3 DISABLED AUTOMATIC RECOVER
PDB2                4 ENABLED
PDB1                5 ENABLED

SQL> !sleep 60

SQL> /
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB$SEED            2 ENABLED
PDB3                3 ENABLED
PDB2                4 ENABLED
PDB1                5 ENABLED
Enter fullscreen mode Exit fullscreen mode

Challenge 2: Standby Restart During Recovery
If the standby is restarted during a PDB Recovery Isolation operation, the process automatically resumes once Data Guard returns to recovery mode.
The datafiles are restored only once.

–Primary

SQL> create pluggable database pdb4 from pdb1;
Pluggable database created.
Enter fullscreen mode Exit fullscreen mode 
–Data Guard

SQL> startup force;

SQL> alter database recover managed standby database disconnect;

SQL> select name,CON_ID,RECOVERY_STATUS from v$pdbs where NAME=’PDB5′;
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB4                6 DISABLED AUTOMATIC RECOVER

SQL> !sleep 60

SQL> /
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB4                6 ENABLED
Enter fullscreen mode Exit fullscreen mode

Challenge 3: Manually Controlling PDB Recovery Isolation
You can cancel an ongoing PDB Recovery Isolation process using:

–Primary

SQL> create pluggable database pdb5 from pdb1;

Pluggable database created.
Enter fullscreen mode Exit fullscreen mode 
–Data Guard

SQL> ALTER PLUGGABLE DATABASE PDB5 RECOVER MANAGED STANDBY DATABASE CANCEL;
Pluggable database altered.

SQL> select name,CON_ID,RECOVERY_STATUS from v$pdbs where NAME=’PDB5′;
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB5                7 DISABLED
Enter fullscreen mode Exit fullscreen mode

To restart it manually:

–Data Guard

SQL> ALTER PLUGGABLE DATABASE PDB5 RECOVER MANAGED STANDBY DATABASE;
Pluggable database altered.

SQL> select name,CON_ID,RECOVERY_STATUS from v$pdbs where NAME=’PDB5′;
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB5                7 DISABLED AUTOMATIC RECOVER

SQL> !sleep 60

SQL> /
NAME           CON_ID RECOVERY_STATUS
———- ———- ————————–
PDB5                7 ENABLED
Enter fullscreen mode Exit fullscreen mode

After about a minute, the RECOVERY_STATUS becomes ENABLED.

Challenge 4: Point-In-Time Recovery (PITR)
As mentioned earlier, in Oracle 21c, PITR at the PDB level is automatically handled by the PDB Recovery Isolation feature.
The following examples demonstrate the behavioral difference between Oracle 19c and Oracle 21c.

– Primary 19c
When performing PITR on a PDB, the Data Guard standby fails with errors such as:

– Primary 19c

[oracle@RAC2 ~]$ rman target /

Recovery Manager: Release 19.0.0.0.0 – Production on Thu Nov 18 16:42:34 2021

Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: DB21C (DBID=249297340)

RMAN>   RUN {
  ALTER PLUGGABLE DATABASE pdb1 CLOSE;
  SET UNTIL SCN 2518869;
  RESTORE PLUGGABLE DATABASE pdb1;
  RECOVER PLUGGABLE DATABASE pdb1;
  ALTER PLUGGABLE DATABASE pdb1 OPEN RESETLOGS;
}

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
———- —————————— ———- ———-
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
Enter fullscreen mode Exit fullscreen mode 
–Data Guard_19c

SQL> alter database recover managed standby database;   
ORA-00283: recovery session canceled due to errors
ORA-65138: Data file 9 of pluggable database PDB1 belongs to an orphan PDB
incarnation.
ORA-01110: data file 9: ‘/oracle19c/base/oradata/DB21C/pdb1/system01.dbf’
Enter fullscreen mode Exit fullscreen mode

– Primary 21c
In contrast, Oracle 21c automatically handles this recovery:

–Primary_21c

SQL> ALTER PLUGGABLE DATABASE pdb1 CLOSE immediate;

Pluggable database altered.

SQL> select name from v$datafile where con_id=3;

[oracle@oLinux7 ~]$ rm -rf /oracle21c/base/oradata/DB21C/D1111E988CA95792E0530488200A2486
Enter fullscreen mode Exit fullscreen mode 
–Data Guard_21c

SQL> ALTER PLUGGABLE DATABASE pdb1 CLOSE immediate;

SQL> select name from v$datafile where con_id=3;

[oracle@oLinux7 ~]$ rm -rf /oracle21c/base/oradata/DB21C/D1111E988CA95792E0530488200A2486
Enter fullscreen mode Exit fullscreen mode 
–Primary_21c

[oracle@oLinux7 ~]$ rman target /

Recovery Manager: Release 21.0.0.0.0 – Production on Thu Nov 18 08:31:59 2021
Version 21.3.0.0.0

Copyright (c) 1982, 2021, Oracle and/or its affiliates.  All rights reserved.

connected to target database: DB21C (DBID=246486353)

RMAN> RUN {
  SET UNTIL SCN 72957710;
  RESTORE PLUGGABLE DATABASE pdb1;
  RECOVER PLUGGABLE DATABASE pdb1;
  ALTER PLUGGABLE DATABASE pdb1 OPEN RESETLOGS;
}

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
———- —————————— ———- ———-
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO

SQL> SELECT con_id, status, pdb_incarnation# inc#, begin_resetlogs_scn, end_resetlogs_scn FROM v$pdb_incarnation ORDER BY 3;
    CON_ID STATUS        INC# BEGIN_RESETLOGS_SCN END_RESETLOGS_SCN
———- ——- ———- ——————- —————–
         3 PARENT           0            27261799          27261799
         3 CURRENT          2            72958406          72958406
Enter fullscreen mode Exit fullscreen mode 
–Data Guard_21c

SQL> select OPEN_MODE from v$database;
OPEN_MODE
——————–
READ ONLY WITH APPLY

SQL> alter pluggable database pdb1 open;
Pluggable database altered.

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
———- —————————— ———- ———-
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ ONLY  NO

SQL> alter session set container=pdb1;
Session altered.

SQL> SELECT con_id, status, pdb_incarnation# inc#, begin_resetlogs_scn, end_resetlogs_scn FROM v$pdb_incarnation ORDER BY 3;
    CON_ID STATUS        INC# BEGIN_RESETLOGS_SCN END_RESETLOGS_SCN
———- ——- ———- ——————- —————–
         3 PARENT           0            27261799          27261799
         3 CURRENT          2            72958406          72958406
Enter fullscreen mode Exit fullscreen mode

Top comments (0)