3 Replies Latest reply: Sep 2, 2013 4:02 AM by user-1221 RSS

    Cannot Start ASM Resource

    user-1221
      Hi,

      After successfully installing PSU 11.2.0.3.6 on Node the Services did not start automatically;

      2 Node RAC on HP-UX 11.31
      $ crsctl stat res -t -init
      --------------------------------------------------------------------------------
      NAME           TARGET  STATE        SERVER                   STATE_DETAILS
      --------------------------------------------------------------------------------
      Cluster Resources
      --------------------------------------------------------------------------------
      ora.asm
            1        ONLINE  OFFLINE
      ora.cluster_interconnect.haip
            1        ONLINE  ONLINE       proddb02
      ora.crsd
            1        ONLINE  ONLINE       proddb02
      ora.cssd
            1        ONLINE  ONLINE       proddb02
      ora.cssdmonitor
            1        ONLINE  ONLINE       proddb02
      ora.ctssd
            1        ONLINE  ONLINE       proddb02                 OBSERVER
      ora.diskmon
            1        OFFLINE OFFLINE
      ora.evmd
            1        ONLINE  ONLINE       proddb02
      ora.gipcd
            1        ONLINE  ONLINE       proddb02
      ora.gpnpd
            1        ONLINE  ONLINE       proddb02
      ora.mdnsd
            1        ONLINE  ONLINE       proddb02
      
      $ crsctl start resource ora.asm
      CRS-2672: Attempting to start 'ora.asm' on proddb02'
      CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
      CRS-5017: The resource action "ora.asm start" encountered the following error:
      ORA-27504: IPC error creating OSD context
      ORA-27300: OS system dependent operation:check if failed with status: 0
      ORA-27301: OS failure message: Error 0
      ORA-27302: failure occurred at: skgxpcini3
      ORA-27303: additional information: requested interface lan1:801 interface not up _disable_interface_checking = TRUE to disable this check for single instance cluster. Check output from ifconfig c
      . For details refer to "(:CLSN00107:)" in "/u01/app/oracle/11.2.0/grid/log/proddb02/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
      
      CRS-2674: Start of 'ora.asm' on 'proddb02' failed
      CRS-2679: Attempting to clean 'ora.asm' on 'proddb02'
      CRS-2681: Clean of 'ora.asm' on 'proddb02' succeeded
      CRS-2674: Start of 'ora.asm' on 'proddb02' failed
      CRS-4000: Command Start failed, or completed with errors.
      
      $ crsctl stat res -t
      --------------------------------------------------------------------------------
      NAME           TARGET  STATE        SERVER                   STATE_DETAILS
      --------------------------------------------------------------------------------
      Local Resources
      --------------------------------------------------------------------------------
      ora.DATADG.dg
                     ONLINE  ONLINE       proddb01
                     ONLINE  OFFLINE      proddb02
      ora.FLASHDG.dg
                     ONLINE  ONLINE       proddb01
                     ONLINE  OFFLINE      proddb02
      ora.LISTENER.lsnr
                     ONLINE  ONLINE       proddb01
                     ONLINE  ONLINE       proddb02
      ora.asm
                     ONLINE  ONLINE       proddb01                 Started
                     ONLINE  OFFLINE      proddb02
      ora.gsd
                     OFFLINE OFFLINE      proddb01
                     OFFLINE OFFLINE      proddb02
      ora.net1.network
                     ONLINE  ONLINE       proddb01
                     ONLINE  ONLINE       proddb02
      ora.ons
                     ONLINE  ONLINE       proddb01
                     ONLINE  ONLINE       proddb02
      --------------------------------------------------------------------------------
      Cluster Resources
      --------------------------------------------------------------------------------
      ora.LISTENER_SCAN1.lsnr
            1        ONLINE  ONLINE       proddb01
      ora.cvu
            1        ONLINE  ONLINE       proddb01
      ora.proddb01.vip
            1        ONLINE  ONLINE       proddb01
      ora.proddb02.vip
            1        ONLINE  ONLINE       proddb02
      ora.oc4j
            1        OFFLINE OFFLINE
      ora.orcl.db
            1        ONLINE  ONLINE       proddb01                 Open
            2        ONLINE  OFFLINE                               Instance Shutdown
      ora.orcl.orcltaf.svc
            1        ONLINE  ONLINE       proddb01
            2        ONLINE  OFFLINE
      ora.scan1.vip
            1        ONLINE  ONLINE       proddb01
      The Interface lan1:801 is not correct, it was not there initially;
      NODE2
      $ oifcfg getif
      lan1  192.168.1.0  global  cluster_interconnect
      lan0  10.241.16.128  global  public
      
      $ oifcfg iflist
      lan1  192.168.1.0
      lan1  169.254.0.0
      lan0  10.141.15.128
      
      NODE1
      $ oifcfg getif
      lan1  192.168.1.0  global  cluster_interconnect
      lan0  10.141.15.128  global  public
      
      $ oifcfg iflist
      lan1  192.168.1.0
      lan0  10.141.15.128
      ORA-27303: additional information: requested interface lan1:801* interface not up disableinterface_checking = TRUE to disable this check for single instance cluster. Check output from ifconfig c*

      Instead of looking for lan1 192.168.1.0 the clusterware is using lan1:801 and this was not there before installing the PSU 11.2.0.3.6.
      may due to that the ASM service is not starting

      Any help would be highly appreciated.
        • 1. Re: Cannot Start ASM Resource
          user-1221
          Trying to remove the virtual lan1:801
          # netstat -inw
          Name              Mtu        Network         Address         Ipkts               Ierrs      Opkts               Oerrs      Coll
          lan1              1500       192.168.1.0     192.168.1.11    69344               0          73890               0          0
          lan0              1500       10.141.15.128   10.141.15.136   7343                0          7293                0          0
          lo0               32808      127.0.0.0       127.0.0.1       250567              0          250567              0          0
          lan1:801          1500       169.254.0.0     169.254.150.220 138                 0          138                 0          0
          lan0:801          1500       10.141.15.128   10.141.15.138   280                 0          0                   0          0
          # ifconfig lan1:801 0.0.0.0 down

          As soon as the interface lan1:801 was down automatically a new interface lan1:802 came into existence with the same ip address
          # netstat -inw
          Name              Mtu        Network         Address         Ipkts               Ierrs      Opkts               Oerrs      Coll
          lan1              1500       192.168.1.0     192.168.1.11    69423               0          73998               0          0
          lan0              1500       10.141.15.128   10.141.15.136   7421                0          7351                0          0
          lo0               32808      127.0.0.0       127.0.0.1       252055              0          252055              0          0
          lan1:801*         1500       none            none            0                   0          0                   0          0
          lan1:802          1500       169.254.0.0     169.254.150.220 0                   0          0                   0          0
          lan0:801          1500       10.141.15.128   10.141.15.138   280                 0          0                   0          0
          # ifconfig lan1:802 0.0.0.0 down

          As soon as the interface lan1:802 was down automatically a new interface lan1:803 came into existence with the same ip address
          # netstat -inw
          Name              Mtu        Network         Address         Ipkts               Ierrs      Opkts               Oerrs      Coll
          lan1              1500       192.168.1.0     192.168.1.11    69491               0          74087               0          0
          lan0              1500       10.141.15.128   10.141.15.136   7437                0          7367                0          0
          lo0               32808      127.0.0.0       127.0.0.1       252208              0          252208              0          0
          lan1:801*         1500       none            none            0                   0          0                   0          0
          lan1:803          1500       169.254.0.0     169.254.150.220 0                   0          0                   0          0
          lan1:802*         1500       none            none            0                   0          0                   0          0
          lan0:801          1500       10.141.15.128   10.141.15.138   280                 0          0                   0          0
          the new virtual interface is getting created once the old one is disabled.

          Brought down the interface lan1:803 and tried to start the "ora.asm" resource

          # ifconfig lan1:803 down
          # netstat -inw
          Name              Mtu        Network         Address         Ipkts               Ierrs      Opkts               Oerrs      Coll
          lan1              1500       192.168.1.0     192.168.1.11    74078               0          79108               0          0
          lan0              1500       10.241.16.128   10.241.16.136   7817                0          7749                0          0
          lo0               32808      127.0.0.0       127.0.0.1       266335              0          266335              0          0
          lan1:801*         1500       none            none            0                   0          0                   0          0
          lan1:803*         1500       none            none            0                   0          0                   0          0
          lan1:802*         1500       none            none            0                   0          0                   0          0
          lan0:801          1500       10.241.16.128   10.241.16.138   314                 0          0                   0          0
          Resource status
          $ crsctl stat res -t -init
          --------------------------------------------------------------------------------
          NAME           TARGET  STATE        SERVER                   STATE_DETAILS
          --------------------------------------------------------------------------------
          Cluster Resources
          --------------------------------------------------------------------------------
          ora.asm
                1        ONLINE  OFFLINE                               Instance Shutdown
          ora.cluster_interconnect.haip
                1        ONLINE  ONLINE       proddb02
          ora.crsd
                1        ONLINE  ONLINE       proddb02
          ora.cssd
                1        ONLINE  ONLINE       proddb02
          ora.cssdmonitor
                1        ONLINE  ONLINE       proddb02
          ora.ctssd
                1        ONLINE  ONLINE       proddb02                 OBSERVER
          ora.diskmon
                1        OFFLINE OFFLINE
          ora.evmd
                1        ONLINE  ONLINE       proddb02
          ora.gipcd
                1        ONLINE  ONLINE       proddb02
          ora.gpnpd
                1        ONLINE  ONLINE       proddb02
          ora.mdnsd
                1        ONLINE  ONLINE       proddb02
          $
          $ crsctl start resource ora.asm
          CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
          CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
          CRS-5017: The resource action "ora.asm start" encountered the following error:
          ORA-27504: IPC error creating OSD context
          ORA-27300: OS system dependent operation:check if failed with status: 0
          ORA-27301: OS failure message: Error 0
          ORA-27302: failure occurred at: skgxpcini3
          ORA-27303: additional information: requested interface lan1:803 interface not up _disable_interface_checking = TRUE to disable this check for single instance cluster. Check output from ifconfig c
          . For details refer to "(:CLSN00107:)" in "/u01/app/oracle/11.2.0/grid/log/proddb02/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
          
          CRS-2674: Start of 'ora.asm' on 'proddb02' failed
          CRS-2679: Attempting to clean 'ora.asm' on 'proddb02'
          CRS-2681: Clean of 'ora.asm' on 'proddb02' succeeded
          CRS-2674: Start of 'ora.asm' on 'proddb02' failed
          CRS-4000: Command Start failed, or completed with errors.
          # ifconfig lan1:804 up
          $ crsctl start resource ora.asm
          CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
          CRS-2672: Attempting to start 'ora.asm' on 'proddb02'
          CRS-5017: The resource action "ora.asm start" encountered the following error:
          ORA-03113: end-of-file on communication channel
          Process ID: 0
          Session ID: 0 Serial number: 0
          . For details refer to "(:CLSN00107:)" in "/u01/app/oracle/11.2.0/grid/log/proddb02/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
          
          CRS-2674: Start of 'ora.asm' on 'proddb02' failed
          CRS-2679: Attempting to clean 'ora.asm' on 'proddb02'
          CRS-2681: Clean of 'ora.asm' on 'proddb02' succeeded
          CRS-2674: Start of 'ora.asm' on proddb02' failed
          CRS-4000: Command Start failed, or completed with errors.
          • 2. Re: Cannot Start ASM Resource
            Ed Rudans

            Just one thing to try. During the PSU apply the relink will be performed. Check for errors in the relink logs.

             

            To confirm that oracle is linked properly

             

            cd $ORACLE_HOME/rdbms/lib

            nm -r libknlopt.a | grep -c kcsm.o

            if returns 0 then then oracle is not linked, need to link it.


            Regards

            Ed

             

            • 3. Re: Cannot Start ASM Resource
              user-1221

              Hi,

              Deconfigure and reconfigure the cluster to resolve the issue