1 Reply Latest reply on Dec 18, 2019 11:10 AM by 4028624

    Solaris / Rac One can't relocate to another node

    4028624

      Hello guys,

       

      i'am running system Oracle Solaris 11.4 SPARC. I've installed Oracle Grid 18, then installed Oracle database 18.3 database software and then created database.

      Everything looked fine after i tried to relocate database to second node.

       

      Before that a run command on node1:

       

      -bash-5.0$ crsctl check crs

      CRS-4638: Oracle High Availability Services is online

      CRS-4537: Cluster Ready Services is online

      CRS-4529: Cluster Synchronization Services is online

      CRS-4533: Event Manager is online

       

      Then:

       

      -bash-5.0$ srvctl relocate database -d IAMDB -n node2

       

      Meanwhile on node2:

       

      -bash-5.0$ srvctl status database -d IAMDB

      Instance IAMDB_1 is running on node node1

      Online relocation: ACTIVE

      Source instance: IAMDB_1 on node1

      Destination instance: IAMDB_2 on node2

       

      But it failed on node1:

       

      PRCD-1222 : Online relocation of database "IAMDB" failed but database was restored to its original state

      PRCD-1129 : Failed to start instance IAMDB_2 for database IAMDB

      PRCR-1064 : Failed to start resource ora.iamdb.db on node2

      CRS-5017: The resource action "ora.iamdb.db start" encountered the following error:

      ORA-03113: end-of-file on communication channel

      Process ID: 0

      Session ID: 0 Serial number: 0

      . For details refer to "(:CLSN00107:)" in "/opt/oraclegrid/base/diag/crs/node2/crs/trace/crsd_oraagent_oracle.trc".

      CRS-2674: Start of 'ora.iamdb.db' on 'node2' failed

       

      I've looked on second node to trc, found:

       

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] ORA-01092: ORACLE instance terminated. Disconnection forced

      Process ID: 0

      Session ID: 0 Serial number: 0

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] InstConnection::connectInt 160 Exception OCIException

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] InstConnection:connect 040 excp OCIException OCI error 1092 count 1

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] InstConnection:connect 100 rethrow excp OCIException OCI error 1092 count 1

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] DbAgent::checkCbk Exception OCIException

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] ORA-01092: ORACLE instance terminated. Disconnection forced

      Process ID: 0

      Session ID: 0 Serial number: 0

       

      In alert log on node2 i found:

       

      OS process OFSD (ospid 17038) idle for 30 seconds, exiting

      2019-12-17T09:12:24.058769+01:00

      LMON received an instance eviction notification from instance 1

      The instance eviction reason is 0x2

      The instance eviction map is 2

      2019-12-17T09:12:30.020612+01:00

      PMON (ospid: 17022): terminating the instance due to ORA error 481

      Cause - 'Instance is being terminated due to fatal process death (pid: 20, ospid: 17051, LMON)'

       

      Please help, thanks.