Skip to Main Content

Oracle Database Discussions

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

AutoUpgrade returns errors on PDB$SEED

User_A7RKTSep 5 2020 — edited Sep 14 2020

Hi guys,

I am upgrading a Logical Standby database from 12c R2 to 19c on Linux 7 using AutoUpgrade.

Every thing went fine until I reached to the point of running the AutoUpgrade on Deploy mode.

The utility worked fine for nearly an hour and ended with the following error:

Errors in database [oradb2]

Stage     [POSTFIXUPS]

Operation [STOPPED]

Status    [ERROR]

Info    [

Error: UPG-1604

Errors executing [alter session set container=PDB$SEED;

Cause: Error executing a database command

For further details, see the log file located at /home/oracle/scripts/autoupgrade/oradb2/102/autoupgrade_20200905_user.log]

-------------------------------------------------

Logs: [/home/oracle/scripts/autoupgrade/oradb2/102/autoupgrade_20200905_user.log]

-------------------------------------------------

The log file reports the following error:

2020-09-05 22:47:24.988 ERROR Dispatcher failed: AutoUpgException [UPG-1603#oracle.upgrade.autoupgrade.utils.errors.AutoUpgException: AutoUpgException [UPG-1604#Errors executing [alter session set container=PDB$SEED;

alter session set "_oracle_script"=true;

alter pluggable database close immediate force instances=all;

alter pluggable database open read write instances=all;

Warning: PDB altered with errors.] [PDB$SEED]]]

2020-09-05 22:47:24.992 INFO Starting error management routine

2020-09-05 22:47:24.997 INFO Ended error management routine

2020-09-05 22:47:24.998 ERROR Error running dispatcher for job 102

Cause: Error executing a database command

2020-09-05 22:47:24.998 ERROR Dispatcher failed:

Error: UPG-1604

Errors executing [alter session set container=PDB$SEED;

The alert log file of the database contains the following error:

PDB$SEED(2):Undo initialization finished serial:0 start:8825641 end:8825809 diff:168 ms (0.2 seconds)

PDB$SEED(2):Database Characterset for PDB$SEED is AL32UTF8

Violations: Type: 1, Count: 1

PDB$SEED(2):***************************************************************

PDB$SEED(2):WARNING: Pluggable Database PDB$SEED with pdb id - 2 is

PDB$SEED(2):         altered with errors or warnings. Please look into

PDB$SEED(2):         PDB_PLUG_IN_VIOLATIONS view for more details.

PDB$SEED(2):***************************************************************

2020-09-05T22:47:24.254992+04:00

PDB$SEED(2):Opening pdb with no Resource Manager plan active

PDB$SEED(2):joxcsys_required_dirobj_exists: directory object exists with required path /u01/app/oracle/product/19.0.0/db_1/javavm/admin/, pid 18241 cid 2

Pluggable database PDB$SEED opened read write

PDB$SEED(2):Completed: alter pluggable database open read write instances=all

2020-09-05T22:59:12.290514+04:00

When I try to login to the database, I see it shutdown.

[oracle@srv1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Sat Sep 5 23:06:43 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount

ORACLE instance started.

Total System Global Area 1073741824 bytes

Fixed Size                  8801008 bytes

Variable Size             440403216 bytes

Database Buffers          616562688 bytes

Redo Buffers                7974912 bytes

ORA-00205: error in identifying control file, check alert log for more info

Any idea on handling the issue?

This post has been answered by User_A7RKT on Sep 7 2020
Jump to Answer

Comments

Wesley D-Oracle

Are you able to login/connect to the Database from the 19c Binaries?  Regarding the Control File(s), do they reside at the location where PFILE/SPFILE references when attempting to startup the DB?

User_A7RKT

>> Are you able to login/connect to the Database from the 19c Binaries? 

Yes

>> Regarding the Control File(s), do they reside at the location where PFILE/SPFILE references when attempting to startup the DB?

Yes

[oracle@srv1 bin]$ pwd

/u01/app/oracle/product/19.0.0/db_1/bin

[oracle@srv1 bin]$ ./sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Sep 6 21:31:44 2020

Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter control_files

NAME                                TYPE        VALUE

------------------------------------ ----------- ------------------------------

control_files                        string      /u01/app/oracle/oradata/ORADB2

                                                /control01.ctl

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

[oracle@srv1 bin]$ ls /u01/app/oracle/oradata/ORADB2/control01.ctl

/u01/app/oracle/oradata/ORADB2/control01.ctl

arvindkumar-Oracle

Could you also share the output of the below query?

set pagesize 1000
select * from PDB_PLUG_IN_VIOLATIONS where status !='RESOLVED';

User_A7RKT

I can't run the query because the db is not open.

SQL> set pagesize 1000

select * from PDB_PLUG_IN_VIOLATIONS where status !='RESOLVED';SQL>

select * from PDB_PLUG_IN_VIOLATIONS where status !='RESOLVED'

              *

ERROR at line 1:

ORA-01219: database or pluggable database not open: queries allowed on fixed

tables or views only

arvindkumar-Oracle

[oracle@srv1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Sat Sep 5 23:06:43 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount

ORACLE instance started.

Total System Global Area 1073741824 bytes

Fixed Size                  8801008 bytes

Variable Size             440403216 bytes

Database Buffers          616562688 bytes

Redo Buffers                7974912 bytes

ORA-00205: error in identifying control file, check alert log for more info

[oracle@srv1 bin]$ pwd

/u01/app/oracle/product/19.0.0/db_1/bin

[oracle@srv1 bin]$ ./sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Sep 6 21:31:44 2020

Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter control_files

NAME                                TYPE        VALUE

------------------------------------ ----------- ------------------------------

control_files                        string      /u01/app/oracle/oradata/ORADB2

                                                /control01.ctl

SQL> exit

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

The above 2 snippet seems to indicate that you are connected to 12.2.0.1.0 version. Can you confirm that the db is started with 19c OH?

I would have expected the db to up after the autoupgrade errored out. You may want to check the alert log and/or "ps -ef|grep pmon" to be sure that the db is started with the correct OH.

User_A7RKT

I issued the following command:

[oracle@srv1 bin]$ ps -ef|grep pmon

..

oracle   17523     1  0 Sep05 ?        00:00:04 ora_pmon_oradb2

oracle   20884     1  0 Sep05 ?        00:00:04 ora_pmon_oradb2

..

The process 17523 is running from 19c home.

The process 20884 is running from 12c home.

I knew it from the command "ps -auxwe | grep 17523". I cannot paste its output because it is too much!

arvindkumar-Oracle

Please shutdown the one running from 12c and try connecting using the instance running with 19c and check the status of the db. If it is open, please run the earlier select statement

User_A7RKT
Answer

Oracle support asked me to perform the following:

cd $ORACLE_HOME/OPatch

./datapatch –verbose

The proposed solution resolved the issue.

Frankly speaking, this is weird. My database is a purely fresh database created by the DBCA. I didn't even install any user data in it. Yet the standard upgrade procedure didn't work on it. I am wondering what kind of testing was done on the new tool.

Marked as Answer by User_A7RKT · Sep 27 2020

Hi,

I am glad that your problem was resolved. Please share the SR number. I would like to dig into the log files to see if we can find out what the root cause of the problem is. Did you attach all the AutoUpgrade log files and the alert log to the service request? If not, it would be very good if you could do so. Thanks in advance.

Regards,

Daniel

User_A7RKT

yes I did provide the files.

But frankly, I didn't like the solution. We want to apply the upgrade using the standard procedure without any manual intervention.

SR No. 3-23962666781

Hi,

Thanks for posting the SR number. I have reached out to the developers for further investigation.

If you want an update on it please e-mail me directly: daniel.overby.hansen@oracle.com.

I don't want to post more details in this public forum, because the details of your SR is private.

Regards,

Daniel

Hi,

Just a final thought. It appears you are upgrading to 19.3.0 which is the base release of 19c. Our clear recommendation is to always upgrade to an Oracle Home that is already patched with the latest Release Update (19.7.0.). A lot of bugs have been fixed since the initial release, so you might be hitting the same problems as other customers because you haven't patched before the upgrade.

Regards,

Daniel

User_A7RKT

I followed your advice. I applied July-2020 RU on the 12c database and on the 19c software. Then tried the upgrade procedure.

It failed with different error this time:

upg>

-------------------------------------------------

Errors in database [oradb2]

Stage     [DBUPGRADE]

Operation [STOPPED]

Status    [ERROR]

Info    [

Error: UPG-1400

UPGRADE FAILED [oradb2]

Cause: Database upgrade failed with errors

For further details, see the log file located at /home/oracle/scripts/autoupgrade/oradb2/102/autoupgrade_20200912_user.log]

In the log file, I can see the following:

2020-09-13 01:26:51.189 ERROR Database Upgrade Error in File [/home/oracle/scripts/autoupgrade/oradb2/102/dbupgrade/catupgrd20200912234841pdb_seed0.log] on Database [/home/oracle/scripts/autoupgrade/oradb2/102/dbupgrade/catupgrd20200912234841pdb_seed0.log]

2020-09-13 01:26:53.740 ERROR

DATABASE NAME: oradb2-PDB$SEED

         CAUSE: ERROR at Line 144781 in [/home/oracle/scripts/autoupgrade/oradb2/102/dbupgrade/catupgrd20200912234841pdb_seed1.log]

        REASON: ORA-03113: end-of-file on communication channel

        ACTION: [MANUAL]

        DETAILS: 03113, 00000, "end-of-file on communication channel"

// *Cause: The connection between Client and Server process was broken.

// *Action: There was a communication error that requires further investigation.

//          First, check for network problems and review the SQL*Net setup.

//          Also, look in the alert.log file for any errors. Finally, test to

//          see whether the server process is dead and whether a trace file

//          was generated at failure time.

2020-09-13 01:26:53.744 ERROR UPGRADE FAILED [oradb2-PDB$SEED]

2020-09-13 01:26:53.744 ERROR Exception Error in Database Upgrade [UPG-1400#UPGRADE FAILED [oradb2-PDB$SEED]]

2020-09-13 01:26:53.744 INFO End Upgrade on Database [oradb2-PDB$SEED]

It looks like I have to create another SR now :-(

User_A7RKT

I dig into the alertlog file and knew the problem was in the sga size. I increased it to 1.5 GB and then all went good.

After applying the patches on 12c and 19c, the issue is resolved.

Thanks Dan!

1 - 14

Post Details

Added on Sep 5 2020
14 comments
2,911 views