This discussion is archived
3 Replies Latest reply: Sep 2, 2013 2:02 AM by user-1221 RSS

Cannot Start ASM Resource

user-1221 Newbie
Currently Being Moderated
Hi,

After successfully installing PSU 11.2.0.3.6 on Node the Services did not start automatically;

2 Node RAC on HP-UX 11.31
$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  OFFLINE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       proddb02
ora.crsd
      1        ONLINE  ONLINE       proddb02
ora.cssd
      1        ONLINE  ONLINE       proddb02
ora.cssdmonitor
      1        ONLINE  ONLINE       proddb02
ora.ctssd
      1        ONLINE  ONLINE       proddb02                 OBSERVER
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       proddb02
ora.gipcd
      1        ONLINE  ONLINE       proddb02
ora.gpnpd
      1        ONLINE  ONLINE       proddb02
ora.mdnsd
      1        ONLINE  ONLINE       proddb02

$ crsctl start resource ora.asm
CRS-2672: Attempting to start 'ora.asm' on proddb02'
CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:check if failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpcini3
ORA-27303: additional information: requested interface lan1:801 interface not up _disable_interface_checking = TRUE to disable this check for single instance cluster. Check output from ifconfig c
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/11.2.0/grid/log/proddb02/agent/ohasd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.asm' on 'proddb02' failed
CRS-2679: Attempting to clean 'ora.asm' on 'proddb02'
CRS-2681: Clean of 'ora.asm' on 'proddb02' succeeded
CRS-2674: Start of 'ora.asm' on 'proddb02' failed
CRS-4000: Command Start failed, or completed with errors.

$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
               ONLINE  ONLINE       proddb01
               ONLINE  OFFLINE      proddb02
ora.FLASHDG.dg
               ONLINE  ONLINE       proddb01
               ONLINE  OFFLINE      proddb02
ora.LISTENER.lsnr
               ONLINE  ONLINE       proddb01
               ONLINE  ONLINE       proddb02
ora.asm
               ONLINE  ONLINE       proddb01                 Started
               ONLINE  OFFLINE      proddb02
ora.gsd
               OFFLINE OFFLINE      proddb01
               OFFLINE OFFLINE      proddb02
ora.net1.network
               ONLINE  ONLINE       proddb01
               ONLINE  ONLINE       proddb02
ora.ons
               ONLINE  ONLINE       proddb01
               ONLINE  ONLINE       proddb02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       proddb01
ora.cvu
      1        ONLINE  ONLINE       proddb01
ora.proddb01.vip
      1        ONLINE  ONLINE       proddb01
ora.proddb02.vip
      1        ONLINE  ONLINE       proddb02
ora.oc4j
      1        OFFLINE OFFLINE
ora.orcl.db
      1        ONLINE  ONLINE       proddb01                 Open
      2        ONLINE  OFFLINE                               Instance Shutdown
ora.orcl.orcltaf.svc
      1        ONLINE  ONLINE       proddb01
      2        ONLINE  OFFLINE
ora.scan1.vip
      1        ONLINE  ONLINE       proddb01
The Interface lan1:801 is not correct, it was not there initially;
NODE2
$ oifcfg getif
lan1  192.168.1.0  global  cluster_interconnect
lan0  10.241.16.128  global  public

$ oifcfg iflist
lan1  192.168.1.0
lan1  169.254.0.0
lan0  10.141.15.128

NODE1
$ oifcfg getif
lan1  192.168.1.0  global  cluster_interconnect
lan0  10.141.15.128  global  public

$ oifcfg iflist
lan1  192.168.1.0
lan0  10.141.15.128
ORA-27303: additional information: requested interface lan1:801* interface not up disableinterface_checking = TRUE to disable this check for single instance cluster. Check output from ifconfig c*

Instead of looking for lan1 192.168.1.0 the clusterware is using lan1:801 and this was not there before installing the PSU 11.2.0.3.6.
may due to that the ASM service is not starting

Any help would be highly appreciated.
  • 1. Re: Cannot Start ASM Resource
    user-1221 Newbie
    Currently Being Moderated
    Trying to remove the virtual lan1:801
    # netstat -inw
    Name              Mtu        Network         Address         Ipkts               Ierrs      Opkts               Oerrs      Coll
    lan1              1500       192.168.1.0     192.168.1.11    69344               0          73890               0          0
    lan0              1500       10.141.15.128   10.141.15.136   7343                0          7293                0          0
    lo0               32808      127.0.0.0       127.0.0.1       250567              0          250567              0          0
    lan1:801          1500       169.254.0.0     169.254.150.220 138                 0          138                 0          0
    lan0:801          1500       10.141.15.128   10.141.15.138   280                 0          0                   0          0
    # ifconfig lan1:801 0.0.0.0 down

    As soon as the interface lan1:801 was down automatically a new interface lan1:802 came into existence with the same ip address
    # netstat -inw
    Name              Mtu        Network         Address         Ipkts               Ierrs      Opkts               Oerrs      Coll
    lan1              1500       192.168.1.0     192.168.1.11    69423               0          73998               0          0
    lan0              1500       10.141.15.128   10.141.15.136   7421                0          7351                0          0
    lo0               32808      127.0.0.0       127.0.0.1       252055              0          252055              0          0
    lan1:801*         1500       none            none            0                   0          0                   0          0
    lan1:802          1500       169.254.0.0     169.254.150.220 0                   0          0                   0          0
    lan0:801          1500       10.141.15.128   10.141.15.138   280                 0          0                   0          0
    # ifconfig lan1:802 0.0.0.0 down

    As soon as the interface lan1:802 was down automatically a new interface lan1:803 came into existence with the same ip address
    # netstat -inw
    Name              Mtu        Network         Address         Ipkts               Ierrs      Opkts               Oerrs      Coll
    lan1              1500       192.168.1.0     192.168.1.11    69491               0          74087               0          0
    lan0              1500       10.141.15.128   10.141.15.136   7437                0          7367                0          0
    lo0               32808      127.0.0.0       127.0.0.1       252208              0          252208              0          0
    lan1:801*         1500       none            none            0                   0          0                   0          0
    lan1:803          1500       169.254.0.0     169.254.150.220 0                   0          0                   0          0
    lan1:802*         1500       none            none            0                   0          0                   0          0
    lan0:801          1500       10.141.15.128   10.141.15.138   280                 0          0                   0          0
    the new virtual interface is getting created once the old one is disabled.

    Brought down the interface lan1:803 and tried to start the "ora.asm" resource

    # ifconfig lan1:803 down
    # netstat -inw
    Name              Mtu        Network         Address         Ipkts               Ierrs      Opkts               Oerrs      Coll
    lan1              1500       192.168.1.0     192.168.1.11    74078               0          79108               0          0
    lan0              1500       10.241.16.128   10.241.16.136   7817                0          7749                0          0
    lo0               32808      127.0.0.0       127.0.0.1       266335              0          266335              0          0
    lan1:801*         1500       none            none            0                   0          0                   0          0
    lan1:803*         1500       none            none            0                   0          0                   0          0
    lan1:802*         1500       none            none            0                   0          0                   0          0
    lan0:801          1500       10.241.16.128   10.241.16.138   314                 0          0                   0          0
    Resource status
    $ crsctl stat res -t -init
    --------------------------------------------------------------------------------
    NAME           TARGET  STATE        SERVER                   STATE_DETAILS
    --------------------------------------------------------------------------------
    Cluster Resources
    --------------------------------------------------------------------------------
    ora.asm
          1        ONLINE  OFFLINE                               Instance Shutdown
    ora.cluster_interconnect.haip
          1        ONLINE  ONLINE       proddb02
    ora.crsd
          1        ONLINE  ONLINE       proddb02
    ora.cssd
          1        ONLINE  ONLINE       proddb02
    ora.cssdmonitor
          1        ONLINE  ONLINE       proddb02
    ora.ctssd
          1        ONLINE  ONLINE       proddb02                 OBSERVER
    ora.diskmon
          1        OFFLINE OFFLINE
    ora.evmd
          1        ONLINE  ONLINE       proddb02
    ora.gipcd
          1        ONLINE  ONLINE       proddb02
    ora.gpnpd
          1        ONLINE  ONLINE       proddb02
    ora.mdnsd
          1        ONLINE  ONLINE       proddb02
    $
    $ crsctl start resource ora.asm
    CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
    CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
    CRS-5017: The resource action "ora.asm start" encountered the following error:
    ORA-27504: IPC error creating OSD context
    ORA-27300: OS system dependent operation:check if failed with status: 0
    ORA-27301: OS failure message: Error 0
    ORA-27302: failure occurred at: skgxpcini3
    ORA-27303: additional information: requested interface lan1:803 interface not up _disable_interface_checking = TRUE to disable this check for single instance cluster. Check output from ifconfig c
    . For details refer to "(:CLSN00107:)" in "/u01/app/oracle/11.2.0/grid/log/proddb02/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    
    CRS-2674: Start of 'ora.asm' on 'proddb02' failed
    CRS-2679: Attempting to clean 'ora.asm' on 'proddb02'
    CRS-2681: Clean of 'ora.asm' on 'proddb02' succeeded
    CRS-2674: Start of 'ora.asm' on 'proddb02' failed
    CRS-4000: Command Start failed, or completed with errors.
    # ifconfig lan1:804 up
    $ crsctl start resource ora.asm
    CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
    CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
    CRS-5017: The resource action "ora.asm start" encountered the following error:
    ORA-03113: end-of-file on communication channel
    Process ID: 0
    Session ID: 0 Serial number: 0
    . For details refer to "(:CLSN00107:)" in "/u01/app/oracle/11.2.0/grid/log/proddb02/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
    
    CRS-2674: Start of 'ora.asm' on 'proddb02' failed
    CRS-2679: Attempting to clean 'ora.asm' on 'proddb02'
    CRS-2681: Clean of 'ora.asm' on 'proddb02' succeeded
    CRS-2674: Start of 'ora.asm' on proddb02' failed
    CRS-4000: Command Start failed, or completed with errors.
  • 2. Re: Cannot Start ASM Resource
    Ed Rudans Explorer
    Currently Being Moderated

    Just one thing to try. During the PSU apply the relink will be performed. Check for errors in the relink logs.

     

    To confirm that oracle is linked properly

     

    cd $ORACLE_HOME/rdbms/lib

    nm -r libknlopt.a | grep -c kcsm.o

    if returns 0 then then oracle is not linked, need to link it.


    Regards

    Ed

     

  • 3. Re: Cannot Start ASM Resource
    user-1221 Newbie
    Currently Being Moderated

    Hi,

    Deconfigure and reconfigure the cluster to resolve the issue

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points