1 Reply Latest reply on Dec 18, 2019 11:10 AM by 4028624

    Solaris / Rac One can't relocate to another node

    4028624
    4028624 Dec 18, 2019 12:39 PM

      Hello guys,

       

      i'am running system Oracle Solaris 11.4 SPARC. I've installed Oracle Grid 18, then installed Oracle database 18.3 database software and then created database.

      Everything looked fine after i tried to relocate database to second node.

       

      Before that a run command on node1:

       

      -bash-5.0$ crsctl check crs

      CRS-4638: Oracle High Availability Services is online

      CRS-4537: Cluster Ready Services is online

      CRS-4529: Cluster Synchronization Services is online

      CRS-4533: Event Manager is online

       

      Then:

       

      -bash-5.0$ srvctl relocate database -d IAMDB -n node2

       

      Meanwhile on node2:

       

      -bash-5.0$ srvctl status database -d IAMDB

      Instance IAMDB_1 is running on node node1

      Online relocation: ACTIVE

      Source instance: IAMDB_1 on node1

      Destination instance: IAMDB_2 on node2

       

      But it failed on node1:

       

      PRCD-1222 : Online relocation of database "IAMDB" failed but database was restored to its original state

      PRCD-1129 : Failed to start instance IAMDB_2 for database IAMDB

      PRCR-1064 : Failed to start resource ora.iamdb.db on node2

      CRS-5017: The resource action "ora.iamdb.db start" encountered the following error:

      ORA-03113: end-of-file on communication channel

      Process ID: 0

      Session ID: 0 Serial number: 0

      . For details refer to "(:CLSN00107:)" in "/opt/oraclegrid/base/diag/crs/node2/crs/trace/crsd_oraagent_oracle.trc".

      CRS-2674: Start of 'ora.iamdb.db' on 'node2' failed

       

      I've looked on second node to trc, found:

       

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] ORA-01092: ORACLE instance terminated. Disconnection forced

      Process ID: 0

      Session ID: 0 Serial number: 0

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] InstConnection::connectInt 160 Exception OCIException

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] InstConnection:connect 040 excp OCIException OCI error 1092 count 1

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] InstConnection:connect 100 rethrow excp OCIException OCI error 1092 count 1

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] DbAgent::checkCbk Exception OCIException

      2019-12-17 09:12:30.431 :CLSDYNAM:19: [ora.iamdb.db]{1:22206:20421} [check] ORA-01092: ORACLE instance terminated. Disconnection forced

      Process ID: 0

      Session ID: 0 Serial number: 0

       

      In alert log on node2 i found:

       

      OS process OFSD (ospid 17038) idle for 30 seconds, exiting

      2019-12-17T09:12:24.058769+01:00

      LMON received an instance eviction notification from instance 1

      The instance eviction reason is 0x2

      The instance eviction map is 2

      2019-12-17T09:12:30.020612+01:00

      PMON (ospid: 17022): terminating the instance due to ORA error 481

      Cause - 'Instance is being terminated due to fatal process death (pid: 20, ospid: 17051, LMON)'

       

      Please help, thanks.