AutoUpgrade returns errors on PDB$SEED

User_A7RKT

Hi guys,

I am upgrading a Logical Standby database from 12c R2 to 19c on Linux 7 using AutoUpgrade.

Every thing went fine until I reached to the point of running the AutoUpgrade on Deploy mode.

The utility worked fine for nearly an hour and ended with the following error:

Errors in database [oradb2]

Stage     [POSTFIXUPS]

Operation [STOPPED]

Status    [ERROR]

Info    [

Error: UPG-1604

Errors executing [alter session set container=PDB$SEED;

Cause: Error executing a database command

For further details, see the log file located at /home/oracle/scripts/autoupgrade/oradb2/102/autoupgrade_20200905_user.log]

-------------------------------------------------

Logs: [/home/oracle/scripts/autoupgrade/oradb2/102/autoupgrade_20200905_user.log]

-------------------------------------------------

The log file reports the following error:

2020-09-05 22:47:24.988 ERROR Dispatcher failed: AutoUpgException [UPG-1603#oracle.upgrade.autoupgrade.utils.errors.AutoUpgException: AutoUpgException [UPG-1604#Errors executing [alter session set container=PDB$SEED;

alter session set "_oracle_script"=true;

alter pluggable database close immediate force instances=all;

alter pluggable database open read write instances=all;

Warning: PDB altered with errors.] [PDB$SEED]]]

2020-09-05 22:47:24.992 INFO Starting error management routine

2020-09-05 22:47:24.997 INFO Ended error management routine

2020-09-05 22:47:24.998 ERROR Error running dispatcher for job 102

Cause: Error executing a database command

2020-09-05 22:47:24.998 ERROR Dispatcher failed:

Error: UPG-1604

Errors executing [alter session set container=PDB$SEED;

The alert log file of the database contains the following error:

PDB$SEED(2):Undo initialization finished serial:0 start:8825641 end:8825809 diff:168 ms (0.2 seconds)

PDB$SEED(2):Database Characterset for PDB$SEED is AL32UTF8

Violations: Type: 1, Count: 1

PDB$SEED(2):***************************************************************

PDB$SEED(2):WARNING: Pluggable Database PDB$SEED with pdb id - 2 is

PDB$SEED(2):         altered with errors or warnings. Please look into

PDB$SEED(2):         PDB_PLUG_IN_VIOLATIONS view for more details.

PDB$SEED(2):***************************************************************

2020-09-05T22:47:24.254992+04:00

PDB$SEED(2):Opening pdb with no Resource Manager plan active

PDB$SEED(2):joxcsys_required_dirobj_exists: directory object exists with required path /u01/app/oracle/product/19.0.0/db_1/javavm/admin/, pid 18241 cid 2

Pluggable database PDB$SEED opened read write

PDB$SEED(2):Completed: alter pluggable database open read write instances=all

2020-09-05T22:59:12.290514+04:00

When I try to login to the database, I see it shutdown.

[[email protected] ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Sat Sep 5 23:06:43 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount

ORACLE instance started.

Total System Global Area 1073741824 bytes

Fixed Size                  8801008 bytes

Variable Size             440403216 bytes

Database Buffers          616562688 bytes

Redo Buffers                7974912 bytes

ORA-00205: error in identifying control file, check alert log for more info

Any idea on handling the issue?

User_A7RKT

Oracle support asked me to perform the following:

cd $ORACLE_HOME/OPatch

./datapatch –verbose

The proposed solution resolved the issue.

Frankly speaking, this is weird. My database is a purely fresh database created by the DBCA. I didn't even install any user data in it. Yet the standard upgrade procedure didn't work on it. I am wondering what kind of testing was done on the new tool.

Wesley D-Oracle

Are you able to login/connect to the Database from the 19c Binaries?  Regarding the Control File(s), do they reside at the location where PFILE/SPFILE references when attempting to startup the DB?

User_A7RKT

>> Are you able to login/connect to the Database from the 19c Binaries? 

Yes

>> Regarding the Control File(s), do they reside at the location where PFILE/SPFILE references when attempting to startup the DB?

Yes

[[email protected] bin]$ pwd

/u01/app/oracle/product/19.0.0/db_1/bin

[[email protected] bin]$ ./sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Sep 6 21:31:44 2020

Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter control_files

NAME                                TYPE        VALUE

------------------------------------ ----------- ------------------------------

control_files                        string      /u01/app/oracle/oradata/ORADB2

                                                /control01.ctl

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

[[email protected] bin]$ ls /u01/app/oracle/oradata/ORADB2/control01.ctl

/u01/app/oracle/oradata/ORADB2/control01.ctl

arvindkumar-Oracle

Could you also share the output of the below query?

set pagesize 1000
select * from PDB_PLUG_IN_VIOLATIONS where status !='RESOLVED';

User_A7RKT

I can't run the query because the db is not open.

SQL> set pagesize 1000

select * from PDB_PLUG_IN_VIOLATIONS where status !='RESOLVED';SQL>

select * from PDB_PLUG_IN_VIOLATIONS where status !='RESOLVED'

              *

ERROR at line 1:

ORA-01219: database or pluggable database not open: queries allowed on fixed

tables or views only

arvindkumar-Oracle

[[email protected] ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Sat Sep 5 23:06:43 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount

ORACLE instance started.

Total System Global Area 1073741824 bytes

Fixed Size                  8801008 bytes

Variable Size             440403216 bytes

Database Buffers          616562688 bytes

Redo Buffers                7974912 bytes

ORA-00205: error in identifying control file, check alert log for more info

[[email protected] bin]$ pwd

/u01/app/oracle/product/19.0.0/db_1/bin

[[email protected] bin]$ ./sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Sep 6 21:31:44 2020

Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter control_files

NAME                                TYPE        VALUE

------------------------------------ ----------- ------------------------------

control_files                        string      /u01/app/oracle/oradata/ORADB2

                                                /control01.ctl

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

The above 2 snippet seems to indicate that you are connected to 12.2.0.1.0 version. Can you confirm that the db is started with 19c OH?

I would have expected the db to up after the autoupgrade errored out. You may want to check the alert log and/or "ps -ef|grep pmon" to be sure that the db is started with the correct OH.

User_A7RKT

I issued the following command:

[[email protected] bin]$ ps -ef|grep pmon

..

oracle   17523     1  0 Sep05 ?        00:00:04 ora_pmon_oradb2

oracle   20884     1  0 Sep05 ?        00:00:04 ora_pmon_oradb2

..

The process 17523 is running from 19c home.

The process 20884 is running from 12c home.

I knew it from the command "ps -auxwe | grep 17523". I cannot paste its output because it is too much!

arvindkumar-Oracle

Please shutdown the one running from 12c and try connecting using the instance running with 19c and check the status of the db. If it is open, please run the earlier select statement

User_A7RKT

Oracle support asked me to perform the following:

cd $ORACLE_HOME/OPatch

./datapatch –verbose

The proposed solution resolved the issue.

Frankly speaking, this is weird. My database is a purely fresh database created by the DBCA. I didn't even install any user data in it. Yet the standard upgrade procedure didn't work on it. I am wondering what kind of testing was done on the new tool.

Daniel Overby Hansen-Oracle

Hi,

I am glad that your problem was resolved. Please share the SR number. I would like to dig into the log files to see if we can find out what the root cause of the problem is. Did you attach all the AutoUpgrade log files and the alert log to the service request? If not, it would be very good if you could do so. Thanks in advance.

Regards,

Daniel

User_A7RKT

yes I did provide the files.

But frankly, I didn't like the solution. We want to apply the upgrade using the standard procedure without any manual intervention.

SR No. 3-23962666781