10 Replies Latest reply: Apr 11, 2012 11:44 PM by baskar.l RSS

    Oracle RAC 2 node architecture-- Node -2 always gets evicted

    user12263161
      Hi,

      I have Oracle RAC DB with simple 2 node architecture( Host RHEL5.5 X 86_64) . The problem we are facing is, whenever there is network failure on either of nodes, always node-2 gets evicted (rebooted). We do not see any abnormal errors on alert.log file on both the nodes.

      The steps followed and results are:

      **Node-1#service network restart**
      **Result: Node-2 evicted**

      **Node-2# service network restart**
      **Result: Node-2 evicted**

      I would like to know why node-1 never gets evicted even if the network is down or restarted on node-1 itself?? Is this normal.

      Regards,
      Raj
        • 1. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
          Pavan Kumar
          Hi,

          Refer to metalink note on understanding node eviction and how to address is Note ID: 219361.1

          - Pavan Kumar N
          • 2. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
            user12263161
            Hi,

            This document only describes, how node eviction works and possible reasons. It does not talk about what to do when always one node is getting evicted.even if the problem is with the other node's network.

            Regards,

            Raj
            • 3. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
              sybrand_b
              As you provide almost no detail at all, and this is a network or interconnect issue, we don't have access to your system, what kind of help do you expect?

              ----------
              Sybrand Bakker
              Senior Oracle DBA
              • 4. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
                user12263161
                Hi,

                Please see the css logs below, which is every time same on node-1 when the network services are down on either of nodes.

                [cssd(11240)]CRS-1612:node prddbs02 (2) at 50% heartbeat fatal, eviction in 14.008 seconds
                2010-06-25 03:41:24.631
                [cssd(11240)]CRS-1612:node prddbs02 (2) at 50% heartbeat fatal, eviction in 13.008 seconds
                2010-06-25 03:41:31.623
                [cssd(11240)]CRS-1611:node prddbs02 (2) at 75% heartbeat fatal, eviction in 6.008 seconds
                2010-06-25 03:41:35.631
                [cssd(11240)]CRS-1610:node prddbs02 (2) at 90% heartbeat fatal, eviction in 2.254 seconds
                2010-06-25 03:41:36.622
                [cssd(11240)]CRS-1610:node prddbs02 (2) at 90% heartbeat fatal, eviction in 1.008 seconds
                2010-06-25 03:41:37.624
                [cssd(11240)]CRS-1610:node prddbs02 (2) at 90% heartbeat fatal, eviction in 0.008 seconds
                2010-06-25 03:41:38.147
                [cssd(11240)]CRS-1607:CSSD evicting node prddbs02. Details in /opt/oracle/product/crs/11.1/log/host1/cssd/ocssd.log.
                [cssd(11240)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prddbs01 .


                Let me know specifically which logs would you need, I will be happy to provide the details :)

                Regards,
                Raj
                • 5. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
                  CKPT
                  >
                  Let me know specifically which logs would you need, I will be happy to provide the details :)
                  post the contents from */opt/oracle/product/crs/11.1/log/host1/cssd/ocssd.log*
                  • 6. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
                    user12263161
                    Hi,

                    Please find the output below:

                    2011-06-03 16:36:02.817: [    CSSD][1216194880]clssnmPollingThread: node prddbs02 (2) at 50% heartbeat fatal, removal in 14.120 seconds
                    2011-06-03 16:36:02.817: [    CSSD][1216194880]clssnmPollingThread: node prddbs02 (2) is impending reconfig, flag 132108, misstime 15880
                    2011-06-03 16:36:02.817: [    CSSD][1216194880]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
                    2011-06-03 16:36:05.994: [    CSSD][1132276032]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 760 > margin 750 cur_ms 1480138014 lastalive 1480137254
                    2011-06-03 16:36:07.493: [    CSSD][1226684736]clssnmSendingThread: sending status msg to all nodes
                    2011-06-03 16:36:07.493: [    CSSD][1226684736]clssnmSendingThread: sent 5 status msgs to all nodes
                    2011-06-03 16:36:08.084: [    CSSD][1132276032]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 850 > margin 750 cur_ms 1480140104 lastalive 1480139254
                    2011-06-03 16:36:09.831: [    CSSD][1216194880]clssnmPollingThread: node prddbs02 (2) at 75% heartbeat fatal, removal in 7.110 seconds
                    2011-06-03 16:36:10.122: [    CSSD][1132276032]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 880 > margin 750 cur_ms 1480142134 lastalive 1480141254
                    2011-06-03 16:36:11.112: [    CSSD][1132276032]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 860 > margin 750 cur_ms 1480143124 lastalive 1480142264
                    2011-06-03 16:36:12.212: [    CSSD][1132276032]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 950 > margin 750 cur_ms 1480144224 lastalive 1480143274
                    2011-06-03 16:36:12.487: [    CSSD][1226684736]clssnmSendingThread: sending status msg to all nodes
                    2011-06-03 16:36:12.487: [    CSSD][1226684736]clssnmSendingThread: sent 5 status msgs to all nodes
                    2011-06-03 16:36:13.840: [    CSSD][1216194880]clssnmPollingThread: local diskTimeout set to 200000 ms, remote disk timeout set to 200000, impending reconfig status(0)
                    :
                    2011-06-03 16:36:14.881: [    CSSD][1205705024]clssgmTagize: version(1), type(13), tagizer(0x494dfe)
                    2011-06-03 16:36:14.881: [    CSSD][1205705024]clssgmHandleDataInvalid: grock HB+ASM, member 2 node 2, birth 21
                    2011-06-03 16:36:17.487: [    CSSD][1226684736]clssnmSendingThread: sending status msg to all nodes
                    2011-06-03 16:36:17.487: [    CSSD][1226684736]clssnmSendingThread: sent 5 status msgs to all nodes
                    2011-06-03 16:36:22.486: [    CSSD][1226684736]clssnmSendingThread: sending status msg to all nodes
                    2011-06-03 16:36:22.486: [    CSSD][1226684736]clssnmSendingThread: sent 5 status msgs to all nodes
                    2011-06-03 16:36:23.162: [ GIPCNET][1205705024]gipcmodNetworkProcessRecv: [network] failed recv attempt endp 0x2eb80c0 [0000000001fed69c] { gipcEndpoint : localAddr 'gipc://prddbs01:80b3-6853-187b-4d2e#192.168.7.1#33842', remoteAddr 'gipc://prddbs02:gm_prddbs-cluster#192.168.7.2#60074', numPend 4, numReady 1, numDone 0, numDead 0, numTransfer 0, objFlags 0x1e10, pidPeer 0, flags 0x2616, usrFlags 0x0 }, req 0x2aaaac308bb0 [0000000001ff4b7d] { gipcReceiveRequest : peerName '', data 0x2aaaac2e3cd8, len 10240, olen 0, off 0, parentEndp 0x2eb80c0, ret gipc
                    2011-06-03 16:36:23.162: [ GIPCNET][1205705024]gipcmodNetworkProcessRecv: slos op : sgipcnTcpRecv
                    2011-06-03 16:36:23.162: [ GIPCNET][1205705024]gipcmodNetworkProcessRecv: slos dep : Connection reset by peer (104)
                    2011-06-03 16:36:23.162: [ GIPCNET][1205705024]gipcmodNetworkProcessRecv: slos loc : recv
                    2011-06-03 16:36:23.162: [ GIPCNET][1205705024]gipcmodNetworkProcessRecv: slos info: dwRet 4294967295, cookie 0x2aaaac308bb0
                    2011-06-03 16:36:23.162: [    CSSD][1205705024]clssgmeventhndlr: Disconnecting endp 0x1fed69c ninf 0x2aaab0000f90
                    2011-06-03 16:36:23.162: [    CSSD][1205705024]clssgmPeerDeactivate: node 2 (prddbs02), death 0, state 0x80000001 connstate 0x1e
                    2011-06-03 16:36:23.162: [GIPCXCPT][1205705024]gipcInternalDissociate: obj 0x2eb80c0 [0000000001fed69c] { gipcEndpoint : localAddr 'gipc://prddbs01:80b3-6853-187b-4d2e#192.168.7.1#33842', remoteAddr 'gipc://prddbs02:gm_prddbs-cluster#192.168.7.2#60074', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x1e10, pidPeer 0, flags 0x261e, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
                    2011-06-03 16:36:32.494: [    CSSD][1226684736]clssnmSendingThread: sent 5 status msgs to all nodes
                    2011-06-03 16:36:37.493: [    CSSD][1226684736]clssnmSendingThread: sending status msg to all nodes
                    2011-06-03 16:36:37.494: [    CSSD][1226684736]clssnmSendingThread: sent 5 status msgs to all nodes
                    2011-06-03 16:36:40.598: [    CSSD][1216194880]clssnmPollingThread: node prddbs02 (2) at 90% heartbeat fatal, removal in 2.870 seconds, seedhbimpd 1
                    2011-06-03 16:36:42.497: [    CSSD][1226684736]clssnmSendingThread: sending status msg to all nodes
                    2011-06-03 16:36:42.497: [    CSSD][1226684736]clssnmSendingThread: sent 5 status msgs to all nodes
                    2011-06-03 16:36:43.476: [    CSSD][1216194880]clssnmPollingThread: Removal started for node prddbs02 (2), flags 0x20000, state 3, wt4c 0
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmDoSyncUpdate: Initiating sync 178830908
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssscUpdateEventValue: NMReconfigInProgress val 1, changes 57
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmDoSyncUpdate: local disk timeout set to 27000 ms, remote disk timeout set to 27000
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmDoSyncUpdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 178830908
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmSetupAckWait: Ack message type (11)
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmSetupAckWait: node(1) is ALIVE
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmSendSync: syncSeqNo(178830908), indicating EXADATA fence initialization complete
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]List of nodes that have ACKed my sync: NULL
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmSendSync: syncSeqNo(178830908)
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmWaitForAcks: Ack message type(11), ackCount(1)
                    2011-06-03 16:36:43.476: [    CSSD][1247664448]clssnmHandleSync: Node prddbs01, number 1, is EXADATA fence capable
                    2011-06-03 16:36:43.476: [    CSSD][1247664448]clssscUpdateEventValue: NMReconfigInProgress val 1, changes 58
                    2011-06-03 16:36:43.476: [    CSSD][1247664448]clssnmHandleSync: local disk timeout set to 27000 ms, remote disk timeout set t:
                    2011-06-03 16:36:43.476: [    CSSD][1247664448]clssnmQueueClientEvent: Sending Event(2), type 2, incarn 178830907
                    2011-06-03 16:36:43.476: [    CSSD][1247664448]clssnmQueueClientEvent: Node[1] state = 3, birth = 178830889, unique = 1305623432
                    2011-06-03 16:36:43.476: [    CSSD][1247664448]clssnmQueueClientEvent: Node[2] state = 5, birth = 178830907, unique = 1307103307
                    2011-06-03 16:36:43.476: [    CSSD][1247664448]clssnmHandleSync: Acknowledging sync: src[1] srcName[prddbs01] seq[73] sync[178830908]
                    2011-06-03 16:36:43.476: [    CSSD][1247664448]clssnmSendAck: node 1, prddbs01, syncSeqNo(178830908) type(11)
                    2011-06-03 16:36:43.476: [    CSSD][1240850064]clssgmStartNMMon: node 1 active, birth 178830889
                    2011-06-03 16:36:43.476: [    CSSD][1247664448]clssnmHandleAck: src[1] dest[1] dom[0] seq[0] sync[178830908] type[11] ackCount(0)
                    2011-06-03 16:36:43.476: [    CSSD][1240850064]clssgmStartNMMon: node 2 active, birth 178830907
                    2011-06-03 16:36:43.476: [    CSSD][1240850064]NMEVENT_SUSPEND [00][00][00][06]
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmSendSync: syncSeqNo(178830908), indicating EXADATA fence initialization complete
                    2011-06-03 16:36:43.476: [    CSSD][1240850064]clssgmUpdateEventValue: CmInfo State val 5, changes 190
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]List of nodes that have ACKed my sync: 1
                    2011-06-03 16:36:43.476: [    CSSD][1240850064]clssgmSuspendAllGrocks: Issue SUSPEND
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmWaitForAcks: done, msg type(11)
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmSetMinMaxVersion:node1 product/protocol (11.2/1.4)
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmSetMinMaxVersion: properties common to all nodes: 1,2,3,4,5,6,7,8,9,10,11,12,13,14
                    2011-06-03 16:36:43.476: [    CSSD][1237174592]clssnmSetMinMaxVersion: min product/protocol (11.2/1.4)
                    2011-06-03 16:36:43.476: [    CSSD][1240850064]clssgmQueueGrockEvent: groupName(IG+ASMSYS$USERS) count(2) master(1) event(2), incarn 22, mbrc 2, to member 1, events 0x0, state 0x0
                    2011-06-03 16:36:43.477: [    CSSD][1237174592]clssnmSetMinMaxVersion: max product/protocol (11.2/1.4)
                    2011-06-03 16:36:43.477: [    CSSD][1237174592]clssnmNeedConfReq: No configuration to change
                    :

                    etc.etc....

                    Let me know if any other logfile required. No unususal messages on /var/log/messages.

                    Regards,
                    Raj
                    • 7. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
                      user12263161
                      2011-06-03 17:12:34.263: [    CSSD][1237174592]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id
                      0, have read a DHB from this node
                      2011-06-03 17:12:34.263: [    CSSD][1237174592]clssnmCheckKillStatus: Node 2, prddbs02, down, LATS(1482295934),timeout(30010)
                      2011-06-03 17:12:34.263: [    CSSD][1237174592]clssnmBldSendUpdate: syncSeqNo(178830910)
                      2011-06-03 17:12:34.263: [    CSSD][1237174592]clssnmBldSendUpdate: using msg version 3
                      2011-06-03 17:12:34.264: [    CSSD][1237174592]clssnmDoSyncUpdate: Sync 178830910 complete!
                      2011-06-03 17:12:34.264: [    CSSD][1237174592]clssnmRcfgMgrThread: Reconfig in progress...
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmHandleUpdate: sync[178830910] src[1], msgvers 3 icin 178830886
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmHandleUpdate: common properties are 1,2,3,4,5,6,7,8,9,10,11,12,13,14
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmUpdateNodeState: node prddbs01, number 1, current state 3, proposed state
                      3, current unique 1305623432, proposed unique 1305623432, prevConuni 0, birth 178830889
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmUpdateNodeState: node prddbs02, number 2, current state 5, proposed state
                      0, current unique 1307104864, proposed unique 1307104864, prevConuni 1307104864, birth 178830909
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmDeactivateNode: node 2, state 5
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmDeactivateNode: node 2 (prddbs02) left cluster
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmSendAck: node 1, prddbs01, syncSeqNo(178830910) type(15)
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssscUpdateEventValue: NMReconfigInProgress val 0, changes 65
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmQueueClientEvent: Sending Event(1), type 1, incarn 178830910
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmQueueClientEvent: Node[1] state = 3, birth = 178830889, unique = 13056234
                      32
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmQueueClientEvent: Node[2] state = 0, birth = 0, unique = 0
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmHandleUpdate: SYNC(178830910) from node(1) completed
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmHandleUpdate: NODE 1 (prddbs01) IS ACTIVE MEMBER OF CLUSTER
                      2011-06-03 17:12:34.264: [    CSSD][1247664448]clssnmHandleUpdate: local disk timeout set to 200000 ms, remote disk timeout se
                      t to 200000
                      2011-06-03 17:12:34.264: [    CSSD][1240850064]clssgmStartNMMon: node 1 active, birth 178830889
                      2011-06-03 17:12:34.264: [    CSSD][1240850064]clssgmStartNMMon: node 2 failed, birth (178830909, 0) (old/new)
                      2011-06-03 17:12:34.264: [    CSSD][1240850064]clssgmUpdateEventValue: Reconfig Event val 1, changes 64
                      2011-06-03 17:12:34.264: [    CSSD][1240850064]clssgmUpdateEventValue: CmInfo State val 3, changes 212
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]clssgmReconfigThread: started for reconfig (178830910)
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]NMEVENT_RECONFIG [00][00][00][02]
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]clssgmWaitOnEventValue: after HoldRequest val 1, eval 1 waited 0
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]clssgmCompareSwapEventValue: changed CmInfo State val 4, from 3, changes 213
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]clssgmCleanupNodeContexts(): cleaning up nodes, rcfg(178830909)
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]clssgmCleanupNodeContexts(): successful cleanup of nodes rcfg(178830910)
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]clssgmUpdateEventValue: ReadyPeers val 1, changes 33
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]clssgmCompareSwapEventValue: changed CmInfo State val 6, from 4, changes 214
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]clssgmEstablishConnections: 1 nodes in cluster incarn 178830910
                      2011-06-03 17:12:34.264: [    CSSD][1279134016]clssgmUpdateEventValue: ConnectedNodes val 0, changes 65
                      2011-06-03 17:12:34.264: [    CSSD][1205705024]clssgmPeerListener: new incarn 178830910. old 178830909
                      2011-06-03 17:12:34.264: [    CSSD][1205705024]clssgmPeerListener: disconn 2 (178830909/178830910) inc 178830909

                      2011-06-03 17:12:34.264: [    CSSD][1205705024]clssgmPeerDeactivate: node 2 (prddbs02), death 178830910, state 0x80000000 connstate 0x1f
                      2011-06-03 17:12:34.264: [GIPCXCPT][1205705024]gipcInternalDissociate: obj 0x2aaaac306ea0 [0000000001ff5d97] { gipcEndpoint : localAddr 'gipc://prddbs01:5ad6-b974-c83f-8a99#192.168.7.1#61185', remoteAddr 'gipc://prddbs02:gm_prddbs-cluster#192.168.7.2#47482', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x1eec, pidPeer 0, flags 0x2616, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
                      2011-06-03 17:12:34.265: [GIPCXCPT][1205705024]gipcDissociateF [clssgmPeerDeactivate : clssgmp.c : 3321]: EXCEPTION[ ret gipcretFail (1) ] failed to dissociate obj 0x2aaaac306ea0 [0000000001ff5d97] { gipcEndpoint : localAddr 'gipc://prddbs01:5ad6-b974-c83f-8a99#192.168.7.1#61185', remoteAddr 'gipc://prddbs02:gm_prddbs-cluster#192.168.7.2#47482', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x1eec, pidPeer 0, flags 0x2616, usrFlags 0x0 }, flags 0x0
                      2011-06-03 17:12:34.265: [    CSSD][1205705024]clssgmCleanFuture: discarded 0 future msgs for 2
                      2011-06-03 17:12:34.265: [    CSSD][1205705024]clssgmPeerListener: connects done (1/1)
                      2011-06-03 17:12:34.265: [    CSSD][1205705024]clssgmUpdateEventValue: ConnectedNodes val 178830910, changes 66
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmWaitChangeEventValue: ev(ConnectedNodes) changed to 178830910
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmEstablishConnections: Sending STATUS message to all nodes for incarnation 178830910
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmPeerBroadcast: msg(0x2f46c58), type(20), len(72)
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmEstablishConnections: (1/1) connected, incarn(178830910)
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmCompareSwapEventValue: changed CmInfo State val 7, from 6, changes 215
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmSetVersions: properties common to all peers: 1,2,3,4,5,6,7,8,9,10
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmEstablishMasterNode: MASTER for 178830910 is node(1) birth(178830889)
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmCompareSwapEventValue: changed CmInfo State val 8, from 7, changes 216
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmMasterCMSync: Synchronizing group/lock status, replay-mode=0
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmCompareSwapEventValue: changed CmInfo State val 9, from 8, changes 217
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmMasterCMSync: processing grock(IG+ASMSYS$USERS) type(2)
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmCleanupOrphanMembers: orphan member(2/IG+ASMSYS$USERS), birth(178830909) on node(2), birth(0/178830910)
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmAllocateRPCIndex: allocated rpc 1300 (0x2aaaaadfa530)
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmRPCBroadcast: rpc(0x514002d), status(0), sendcount(0), filtered by specific properties: 8
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmFreeRPCIndex: freeing rpc 1300
                      2011-06-03 17:12:34.265: [    CSSD][1279134016]clssgmCopyoutMemberInfo: IG+ASMSYS$USERS, id 10, gin 1 grp priv data 0, members:
                      • 8. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
                        930002
                        This is normal behaviour as oracle Rac will always keep the lower number of node when there is an issue with private connectivity and will evacuate the bigger node number, in your case node2.
                        • 9. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
                          onedbguru
                          I would not worry so much about which node gets evicted, but why does ANY node get evicted. I have had clusters go for years without node evictions. You have a NIC or network problem. You need to stop focusing on the symptom and fix the actual problem.

                          What is your network connectivity for the interconnect? Hopefully you are using a stand-alone switch for the interconnect with NO other connections. If you are using a director-class switch and using VLANS, it is possible to flood the switch to the extent that the cluster-interconnect is unable to communicate causing the failure. Again, you need to fix the PROBLEM and not the SYMPTOM.
                          • 10. Re: Oracle RAC 2 node architecture-- Node -2 always gets evicted
                            baskar.l
                            Hi,

                            You can use Oracle Instantaneous Problem Detection Tool for OS (IPD/OS) to detect and analyze OS and cluster resource degradation and failures.

                            Thanks,
                            baskar.l