This discussion is archived
7 Replies Latest reply: Apr 16, 2013 1:01 PM by 1003436 RSS

Multiple NIC HAIP problem ORA-27301: .. Network is unreachable 11.2.0.3.4

ujang jaenudin Newbie
Currently Being Moderated
dear all,

I just configure RAC 11.2.0.3.4. on solaris 11
Installation just fine, running root.sh without issue.

During installation I set only 1 NIC for interconnect with this ip 10.110.86.17/29 (node1), 10.110.86.18/29 (node2), 10.110.86.19/29 (node3).

later on after create database and import from uat database, adding 3 more NIC for interconnect, but this error messages keep coming at ASM alert log on all 3 nodes.

Errors in file /apps/oracle/diag/rdbms/opasm/OPASM3/trace/OPASM3_dia0_11800.trc:
ORA-27508: IPC error sending a message
ORA-27300: OS system dependent operation:sendmsg failed with status: 128
ORA-27301: OS failure message: Network is unreachable
ORA-27302: failure occurred at: sskgxpsnd2


node1:
10.110.86.17/29
10.110.87.17/29
10.110.88.17/29
10.110.89.17/29

node2:
10.110.86.18/29
10.110.87.18/29
10.110.88.18/29
10.110.89.18/29

node3:
10.110.86.19/29
10.110.87.19/29
10.110.88.19/29
10.110.89.19/29


looking at orarootagent root of ohasd log, these text appear:


2013-02-26 21:11:21.025: [ USRTHRD][18] {0:0:2} Reassigned IP: 169.254.217.161 on interface net5
2013-02-26 21:11:21.026: [ USRTHRD][18] {0:0:2} USING HAIP[  0 ]: net4 - 169.254.141.133
2013-02-26 21:11:21.026: [ USRTHRD][18] {0:0:2} USING HAIP[  1 ]: net1 - 169.254.2.193
2013-02-26 21:11:21.026: [ USRTHRD][18] {0:0:2} USING HAIP[  2 ]: net5 - 169.254.217.161
2013-02-26 21:11:21.026: [ USRTHRD][18] {0:0:2} USING HAIP[  3 ]: net2 - 169.254.84.149
2013-02-26 21:11:23.976: [ora.crsd][17] {0:0:2} [check] clsdmc_respget return: status=0, ecode=20201
2013-02-26 21:11:23.976: [ora.crsd][17] {0:0:2} [check] DaemonAgent::check returned 0
[  clsdmc][17]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=1B7F958



2013-02-26 21:15:12.442: [ora.ctssd][30] {0:0:2} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[101b7ed38], buflen=8
2013-02-26 21:15:12.442: [ora.ctssd][30] {0:0:2} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [101b7ed38]: mode[0xcc] offset[0 ms].
2013-02-26 21:15:17.408: [ USRTHRD][18] {0:0:2} HAIP: Moving ip '169.254.141.133' from inf 'net4' to inf 'net5'
2013-02-26 21:15:17.408: [ USRTHRD][18] {0:0:2} pausing thread
2013-02-26 21:15:17.408: [ USRTHRD][18] {0:0:2} posting thread
2013-02-26 21:15:17.408: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]start {
2013-02-26 21:15:17.408: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]start }
2013-02-26 21:15:17.408: [ USRTHRD][18] {0:0:2} HAIP: Moving ip '169.254.217.161' from inf 'net5' to inf 'net8'
2013-02-26 21:15:17.408: [ USRTHRD][66] {0:0:2} [NetHAWork] thread started
2013-02-26 21:15:17.408: [ USRTHRD][18] {0:0:2} pausing thread
2013-02-26 21:15:17.408: [ USRTHRD][66] {0:0:2} Arp::sCreateSocket {
2013-02-26 21:15:17.408: [ USRTHRD][18] {0:0:2} posting thread
2013-02-26 21:15:17.408: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]start {
2013-02-26 21:15:17.409: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]start }
2013-02-26 21:15:17.409: [ USRTHRD][67] {0:0:2} [NetHAWork] thread started
2013-02-26 21:15:17.409: [ USRTHRD][67] {0:0:2} Arp::sCreateSocket {
2013-02-26 21:15:17.410: [ USRTHRD][66] {0:0:2} Arp::sCreateSocket }
2013-02-26 21:15:17.410: [ USRTHRD][67] {0:0:2} Arp::sCreateSocket }
2013-02-26 21:15:17.410: [ USRTHRD][66] {0:0:2} Starting Probe for ip 169.254.141.133
2013-02-26 21:15:17.410: [ USRTHRD][66] {0:0:2} Transitioning to Probe State
2013-02-26 21:15:17.411: [ USRTHRD][67] {0:0:2} Starting Probe for ip 169.254.217.161
2013-02-26 21:15:17.411: [ USRTHRD][67] {0:0:2} Transitioning to Probe State
2013-02-26 21:15:17.572: [ USRTHRD][67] {0:0:2} Arp::sProbe {
2013-02-26 21:15:17.572: [ USRTHRD][67] {0:0:2} Arp::sSend: sending type 1
2013-02-26 21:15:17.572: [ USRTHRD][67] {0:0:2} Arp::sProbe }
2013-02-26 21:15:17.773: [ USRTHRD][66] {0:0:2} Arp::sProbe {
2013-02-26 21:15:17.773: [ USRTHRD][66] {0:0:2} Arp::sSend: sending type 1
2013-02-26 21:15:17.773: [ USRTHRD][66] {0:0:2} Arp::sProbe }
2013-02-26 21:15:18.788: [ USRTHRD][66] {0:0:2} Arp::sProbe {
2013-02-26 21:15:18.788: [ USRTHRD][66] {0:0:2} Arp::sSend: sending type 1
2013-02-26 21:15:18.788: [ USRTHRD][66] {0:0:2} Arp::sProbe }
2013-02-26 21:15:19.300: [ USRTHRD][67] {0:0:2} Arp::sProbe {
2013-02-26 21:15:19.301: [ USRTHRD][67] {0:0:2} Arp::sSend: sending type 1
2013-02-26 21:15:19.301: [ USRTHRD][67] {0:0:2} Arp::sProbe }
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} HAIP: Moving ip '' from inf 'net5' to inf 'net4'
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} pausing thread
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} posting thread
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]start {
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]start }
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} HAIP: Moving ip '' from inf 'net8' to inf 'net5'
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} pausing thread
2013-02-26 21:15:19.405: [ USRTHRD][68] {0:0:2} [NetHAWork] thread started
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} posting thread
2013-02-26 21:15:19.405: [ USRTHRD][68] {0:0:2} Arp::sCreateSocket {
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]start {
2013-02-26 21:15:19.405: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]start }
2013-02-26 21:15:19.405: [ USRTHRD][69] {0:0:2} [NetHAWork] thread started
2013-02-26 21:15:19.405: [ USRTHRD][69] {0:0:2} Arp::sCreateSocket {
2013-02-26 21:15:19.406: [ USRTHRD][68] {0:0:2} Arp::sCreateSocket }
2013-02-26 21:15:19.406: [ USRTHRD][69] {0:0:2} Arp::sCreateSocket }
2013-02-26 21:15:19.407: [ USRTHRD][68] {0:0:2} Found existing ip 169.254.141.133 for interface net4, transitioning to DEFEND state
2013-02-26 21:15:19.408: [ USRTHRD][69] {0:0:2} Found existing ip 169.254.217.161 for interface net5, transitioning to DEFEND state
2013-02-26 21:15:19.408: [ USRTHRD][68] {0:0:2} Reattached IP: 169.254.141.133 on interface net4
2013-02-26 21:15:19.408: [ USRTHRD][69] {0:0:2} Reattached IP: 169.254.217.161 on interface net5
2013-02-26 21:15:19.989: [ora.crf][30] {0:0:2} [check] clsdmc_respget return: status=0, ecode=0
2013-02-26 21:15:19.989: [ora.crf][30] {0:0:2} [check] Check return = 0, state detail = NULL
2013-02-26 21:15:21.412: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]stop {
2013-02-26 21:15:21.483: [ USRTHRD][60] {0:0:2} [NetHAWork] thread stopping
2013-02-26 21:15:21.483: [ USRTHRD][60] {0:0:2} Thread:[NetHAWork]isRunning is reset to false here
2013-02-26 21:15:21.484: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]stop }
2013-02-26 21:15:21.484: [ USRTHRD][18] {0:0:2} VipActions::stopIp {
2013-02-26 21:15:21.484: [ USRTHRD][18] {0:0:2} NetInterface::sStopIp {
2013-02-26 21:15:21.484: [ USRTHRD][18] {0:0:2} Stopping ip '169.254.141.133', inf 'net4', mask '10.110.88.16'
2013-02-26 21:15:21.519: [ USRTHRD][18] {0:0:2} NetInterface::sStopIp }
2013-02-26 21:15:21.519: [ USRTHRD][18] {0:0:2} VipActions::stopIp }
2013-02-26 21:15:21.520: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]stop {
2013-02-26 21:15:21.844: [ USRTHRD][66] {0:0:2} [NetHAWork] thread stopping
2013-02-26 21:15:21.844: [ USRTHRD][66] {0:0:2} Thread:[NetHAWork]isRunning is reset to false here
2013-02-26 21:15:21.845: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]stop }
2013-02-26 21:15:21.845: [ USRTHRD][18] {0:0:2} Assigned IP 169.254.141.133 no longer valid on inf net4
2013-02-26 21:15:21.845: [ USRTHRD][18] {0:0:2} VipActions::startIp {
2013-02-26 21:15:21.845: [ USRTHRD][18] {0:0:2} Adding 169.254.141.133 on net4
2013-02-26 21:15:22.915: [ USRTHRD][18] {0:0:2} VipActions::startIp }
2013-02-26 21:15:22.915: [ USRTHRD][18] {0:0:2} Reassigned IP: 169.254.141.133 on interface net4
2013-02-26 21:15:22.916: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]stop {
2013-02-26 21:15:22.986: [ USRTHRD][61] {0:0:2} [NetHAWork] thread stopping
2013-02-26 21:15:22.986: [ USRTHRD][61] {0:0:2} Thread:[NetHAWork]isRunning is reset to false here
2013-02-26 21:15:22.986: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]stop }
2013-02-26 21:15:22.986: [ USRTHRD][18] {0:0:2} VipActions::stopIp {
2013-02-26 21:15:22.986: [ USRTHRD][18] {0:0:2} NetInterface::sStopIp {
2013-02-26 21:15:22.986: [ USRTHRD][18] {0:0:2} Stopping ip '169.254.217.161', inf 'net5', mask '10.110.89.16'
2013-02-26 21:15:23.022: [ USRTHRD][18] {0:0:2} NetInterface::sStopIp }
2013-02-26 21:15:23.022: [ USRTHRD][18] {0:0:2} VipActions::stopIp }
2013-02-26 21:15:23.022: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]stop {
2013-02-26 21:15:23.181: [ USRTHRD][69] {0:0:2} Assigned IP 169.254.217.161 no longer valid on inf net5
2013-02-26 21:15:23.181: [ USRTHRD][69] {0:0:2} VipActions::startIp {
2013-02-26 21:15:23.181: [ USRTHRD][69] {0:0:2} Adding 169.254.217.161 on net5
2013-02-26 21:15:23.379: [ USRTHRD][67] {0:0:2} [NetHAWork] thread stopping
2013-02-26 21:15:23.379: [ USRTHRD][67] {0:0:2} Thread:[NetHAWork]isRunning is reset to false here
2013-02-26 21:15:23.379: [ USRTHRD][18] {0:0:2} Thread:[NetHAWork]stop }
2013-02-26 21:15:23.380: [ USRTHRD][18] {0:0:2} USING HAIP[  0 ]: net4 - 169.254.141.133
2013-02-26 21:15:23.380: [ USRTHRD][18] {0:0:2} USING HAIP[  1 ]: net1 - 169.254.2.193
2013-02-26 21:15:23.380: [ USRTHRD][18] {0:0:2} USING HAIP[  2 ]: net5 - 169.254.217.161
2013-02-26 21:15:23.380: [ USRTHRD][18] {0:0:2} USING HAIP[  3 ]: net2 - 169.254.84.149
2013-02-26 21:15:24.250: [ USRTHRD][69] {0:0:2} VipActions::startIp }
2013-02-26 21:15:24.250: [ USRTHRD][69] {0:0:2} Reassigned IP: 169.254.217.161 on interface net5
2013-02-26 21:15:25.978: [ora.crsd][34] {0:5:6} [check] clsdmc_respget return: status=0, ecode=10201
2013-02-26 21:15:25.978: [ora.crsd][34] {0:5:6} [check] DaemonAgent::check returned 0
2013-02-26 21:15:25.979: [    AGFW][10] {0:5:6} ora.crsd 1 1 state changed from: PARTIAL to: ONLINE
2013-02-26 21:15:25.979: [    AGFW][10] {0:6:11} Generating new Tint for unplanned state change. Original Tint: {0:5:6}
2013-02-26 21:15:25.979: [    AGFW][10] {0:6:11} Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:665
[  clsdmc][34]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=23DB8D8
2013-02-26 21:15:42.442: [ora.ctssd][34] {0:0:2} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[1023db8d8], buflen=8
2013-02-26 21:15:42.442: [ora.ctssd][34] {0:0:2} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [1023db8d8]: mode[0xcc] offset[0 ms].
2013-02-26 21:15:48.715: [    AGFW][10] {0:0:2} Agent received the message: AGENT_HB[Engine] ID 12293:594

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points