13 Replies Latest reply: May 24, 2013 4:12 AM by Red8 RSS

    unable to start asm on one of the nodes

    Red8
      Dear Friends

      due to some problem with our AC all servers in our datacenter where powered off, and after all is fixed and try to start the RAC, it is showing the following errors,
      can you please help me


      [root@KIFIYA-UBS-ORA-RAC1 bin]# ./crsctl start cluster
      CRS-2672: Attempting to start 'ora.asm' on 'kifiya-ubs-ora-rac1'
      ORA-01031: insufficient privileges
      CRS-5017: The resource action "ora.asm start" encountered the following error:
      ORA-01031: insufficient privileges
      . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".
      CRS-2674: Start of 'ora.asm' on 'kifiya-ubs-ora-rac1' failed
      CRS-2679: Attempting to clean 'ora.asm' on 'kifiya-ubs-ora-rac1'
      ORA-01031: insufficient privileges
      CRS-2681: Clean of 'ora.asm' on 'kifiya-ubs-ora-rac1' succeeded
      CRS-4000: Command Start failed, or completed with errors.
        • 1. Re: unable to start asm on one of the nodes
          AshishShukla
          Hello,

          I guess you are using 11gR2 RAC in your environment. So the problem is related with kernel parameters *"/etc/sysctl.conf"*

          Actually sometimes if we are using *"RunFixup"* at the time of installation to modify kernel parameters then this kind of problems occurs.

          So finally, solution is modify kernel parameter file on that machine manually. Hope it will work.


          Thanks,
          Ashish Shukla
          • 2. Re: unable to start asm on one of the nodes
            Red8
            thanks for your response,

            yes i am using 11gR2.0.3 RAC with red hat 6.3 and it has been working for the last 3 month, all was ok, installation went smoothly

            but know after the restart this node is creating problems,

            kernel parameters are fine also, so what could be the problem ?


            here is the trace file output, may be Unable to get log name be the problem cos most of the trace file is filled with this

            2013-05-05 03:04:31.067: [ default]failed to initialize skgp context
            2013-05-05 03:04:31.067: [ default]slos op : sslssreghdlr
            2013-05-05 03:04:31.067: [ default]slos dep : Error 0 (0)
            2013-05-05 03:04:31.067: [ default]slos loc : sskgpinit1
            2013-05-05 03:04:31.067: [ default]slos info:
            [   CLWAL]clsw_Initialize: OLR initlevel [30000]
            2013-05-05 03:04:31.069: [ default]a_init: Unable to get log name. Retval:[-4]
            2013-05-05 03:04:31.096: [    GPNP]clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:313] Result: (0) CLSGPNP_OK. got ASM-Profile.DiscoveryString='/dev/mapper/*'
            kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

            *** 2013-05-16 08:59:48.294
            kfgbUnbind: unbinding group 2/0xab2a4083 (FRA_DATA)
            kfgbUnregister: unregistering group 2/0xab2a4083 (FRA_DATA)

            *** 2013-05-16 08:59:58.305
            kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
            kfdp_query(DATA): 23
            ----- Abridged Call Stack Trace -----
            ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
            <-__libc_start_main()+253<-_start()+36                                                                                                              
            ----- End of Abridged Call Stack Trace -----

            *** 2013-05-16 09:00:08.690
            kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

            *** 2013-05-16 09:00:18.701
            kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
            kfdp_query(DATA): 24
            ----- Abridged Call Stack Trace -----
            ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
            <-__libc_start_main()+253<-_start()+36                                                                                                              
            ----- End of Abridged Call Stack Trace -----

            *** 2013-05-16 09:00:28.901
            kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

            *** 2013-05-16 09:00:38.913
            kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
            kfdp_query(DATA): 25
            ----- Abridged Call Stack Trace -----
            ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
            <-__libc_start_main()+253<-_start()+36
            ----- End of Abridged Call Stack Trace -----

            *** 2013-05-16 09:00:49.109
            kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

            *** 2013-05-16 09:00:59.120
            kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
            kfdp_query(DATA): 26
            ----- Abridged Call Stack Trace -----
            ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
            <-__libc_start_main()+253<-_start()+36
            ----- End of Abridged Call Stack Trace -----

            *** 2013-05-16 09:01:09.353
            kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

            *** 2013-05-16 09:01:19.363
            kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
            kfdp_query(DATA): 27
            ----- Abridged Call Stack Trace -----
            ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
            <-__libc_start_main()+253<-_start()+36
            ----- End of Abridged Call Stack Trace -----

            Edited by: Red8 on May 16, 2013 9:34 PM
            • 3. Re: unable to start asm on one of the nodes
              AshishShukla
              Yes, it happens when you will restart your machines kernel parameters start misbehaving if you have updated kernel parameter with OUI runfixup.

              Modify kernel parameters again manually.

              fs.suid_dumpable = 1
              fs.aio-max-nr = 1048576
              fs.file-max = 6815744
              kernel.shmall = 2097152
              kernel.shmmax = 536870912
              kernel.shmmni = 4096
              kernel.sem = 250 32000 100 128
              net.ipv4.ip_local_port_range = 9000 65500
              net.core.rmem_default = 262144
              net.core.rmem_max = 4194304
              net.core.wmem_default = 262144
              net.core.wmem_max = 1048586
              • 4. Re: unable to start asm on one of the nodes
                Red8
                Thanks for your help, i really apperciate it

                This are what i have in my /etc/sysctl.conf.

                fs.file-max = 6815744
                net.ipv4.ip_local_port_range = 9000 65500
                net.core.rmem_default = 262144
                net.core.rmem_max = 4194304
                net.core.wmem_default = 262144
                net.core.wmem_max = 1048576
                kernel.shmall = 8189848
                kernel.shmmax = 33545619456
                kernel.shmmni = 4096
                kernel.sem = 250 32000 100 128
                • 5. Re: unable to start asm on one of the nodes
                  AshishShukla
                  Optionally do can do one more thing, Copy "/etc/sysctl.conf" file from the node where ASM is running and paste to the problematic machine.

                  # systcl -p

                  and then try to start ASM again. If still problem is there then please share "Clusterware Alert log file" and "crsd.log" files.


                  Thanks,
                  Ashish Shukla
                  • 6. Re: unable to start asm on one of the nodes
                    Red8
                    I have copied the /etc/sysctl.conf file from the working node to the failed node ad tried to restart the asm instance, but it is not working.

                    so i please take a look at the logs

                    1.Clusterware Alert log file

                    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
                    2013-05-16 23:58:44.864
                    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
                    2013-05-17 00:29:33.443
                    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
                    2013-05-17 00:59:57.993
                    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
                    2013-05-17 01:13:20.965
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 01:13:21.194
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 01:13:21.375
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 01:13:21.601
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 01:31:10.556
                    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
                    2013-05-17 01:36:06.063
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 01:36:06.284
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 01:36:06.466
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 01:36:06.688
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 02:01:59.110
                    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
                    2013-05-17 02:15:16.234
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 02:15:16.462
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 02:15:16.643
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 02:15:16.862
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 02:29:11.423
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 02:29:11.651
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 02:29:11.833
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 02:29:12.055
                    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
                    2013-05-17 02:32:07.671
                    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
                    • 7. Re: unable to start asm on one of the nodes
                      Red8
                      2. crsd.log

                      [root@KIFIYA-UBS-ORA-RAC1 bin]# cat /u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/crsd/crsd.log

                      2013-05-17 10:53:01.713: [ default][2997434144] First attempt: init CSS context succeeded.
                      2013-05-17 10:53:01.713: [ default][2997434144] CRS Daemon Starting
                      2013-05-17 10:53:01.714: [ CRSMAIN][2997434144] Checking the OCR device
                      2013-05-17 10:53:01.715: [ CRSMAIN][2997434144] Sync-up with OCR
                      2013-05-17 10:53:01.715: [ CRSMAIN][2997434144] Connecting to the CSS Daemon
                      2013-05-17 10:53:01.715: [ CRSMAIN][2997434144] Getting local node number
                      2013-05-17 10:53:01.715: [ CRSMAIN][2997434144] Initializing OCR
                      [  clsdmt][2982430464]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=kifiya-ubs-ora-rac1DBG_CRSD))
                      2013-05-17 10:53:01.717: [  clsdmt][2982430464]PID for the Process [14363], connkey 1
                      2013-05-17 10:53:01.717: [  clsdmt][2982430464]Creating PID [14363] file for home /u01/app/11.2.0/grid host kifiya-ubs-ora-rac1 bin crs to /u01/app/11.2.0/grid/crs/init/
                      2013-05-17 10:53:01.717: [  clsdmt][2982430464]Writing PID [14363] to the file [u01/app/11.2.0/grid/crs/init/kifiya-ubs-ora-rac1.pid]
                      [   CLWAL][2997434144]clsw_Initialize: OLR initlevel [70000]
                      2013-05-17 10:53:04.012: [  OCRASM][2997434144]proprasmo: Error in open/create file in dg [DATA]
                      [  OCRASM][2997434144]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

                      2013-05-17 10:53:04.012: [  OCRASM][2997434144]ASM Error Stack : ORA-15077: could not locate ASM instance serving a required diskgroup

                      2013-05-17 10:53:04.015: [  OCRASM][2997434144]proprasmo: kgfoCheckMount returned [7]
                      2013-05-17 10:53:04.015: [  OCRASM][2997434144]proprasmo: The ASM instance is down
                      2013-05-17 10:53:04.015: [  OCRRAW][2997434144]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
                      2013-05-17 10:53:04.015: [  OCRRAW][2997434144]proprioo: No OCR/OLR devices are usable
                      2013-05-17 10:53:04.015: [  OCRASM][2997434144]proprasmcl: asmhandle is NULL
                      2013-05-17 10:53:04.016: [    GIPC][2997434144] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 690], original from [clsss.c : 5326]
                      2013-05-17 10:53:04.017: [ default][2997434144]clsvactversion:4: Retrieving Active Version from local storage.
                      2013-05-17 10:53:04.020: [  OCRRAW][2997434144]proprrepauto: The local OCR configuration matches with the configuration published by OCR Cache Writer. No repair required.
                      2013-05-17 10:53:04.022: [  OCRRAW][2997434144]proprinit: Could not open raw device
                      2013-05-17 10:53:04.022: [  OCRASM][2997434144]proprasmcl: asmhandle is NULL
                      2013-05-17 10:53:04.024: [  OCRAPI][2997434144]a_init:16!: Backend init unsuccessful : [26]
                      2013-05-17 10:53:04.025: [  CRSOCR][2997434144] OCR context init failure. Error: PROC-26: Error while accessing the physical storage
                      ORA-15077: could not locate ASM instance serving a required diskgroup

                      2013-05-17 10:53:04.025: [ CRSMAIN][2997434144] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-26: Error while accessing the physical storage
                      ORA-15077: could not locate ASM instance serving a required diskgroup

                      2013-05-17 10:53:04.025: [    CRSD][2997434144][PANIC] CRSD exiting: Could not init OCR, code: 26
                      2013-05-17 10:53:04.025: [    CRSD][2997434144] Done.

                      [root@KIFIYA-UBS-ORA-RAC1 bin]#
                      [root@KIFIYA-UBS-ORA-RAC1 bin]#
                      [root@KIFIYA-UBS-ORA-RAC1 bin]# ls -l /dev/mapper/
                      total 0
                      crw-rw---- 1 root root 10, 58 May 17 10:16 control
                      lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_DATA1 -> ../dm-2
                      lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_DATA2 -> ../dm-8
                      lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_FRA1 -> ../dm-7
                      lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_FRA2 -> ../dm-6
                      lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK1 -> ../dm-0
                      lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK2 -> ../dm-3
                      lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK3 -> ../dm-5
                      lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK4 -> ../dm-4
                      lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK5 -> ../dm-1

                      [root@KIFIYA-UBS-ORA-RAC1 bin]# ./crsctl query css votedisk
                      ## STATE File Universal Id File Name Disk group
                      -- ----- ----------------- --------- ---------
                      1. ONLINE 980d4cd4601e4f99bfa880f99a2ba34b (/dev/mapper/VOTE_DSK5) [DATA]
                      2. ONLINE 7dc793a3ba6b4f5dbf7387f6644f46da (/dev/mapper/VOTE_DSK4) [DATA]
                      3. ONLINE 9931f77f96044f15bf779f8ebe0eb028 (/dev/mapper/VOTE_DSK3) [DATA]
                      4. ONLINE a1870854f20e4f49bf514541b9af9d29 (/dev/mapper/VOTE_DSK2) [DATA]
                      5. ONLINE 15999290b89f4f89bf0b27edc9079ec7 (/dev/mapper/VOTE_DSK1) [DATA]
                      Located 5 voting disk(s).
                      [root@KIFIYA-UBS-ORA-RAC1 bin]

                      Edited by: Red8 on May 17, 2013 1:49 AM
                      • 8. Re: unable to start asm on one of the nodes
                        AshishShukla
                        please share this log file:

                        "*/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".*
                        • 9. Re: unable to start asm on one of the nodes
                          Red8
                          2013-05-17 01:13:13.720: [ora.mdnsd][3477051136] {0:0:2} [check] clsdmc_respget return: status=0, ecode=0
                          2013-05-17 01:13:15.345: [ora.gpnpd][3477051136] {0:0:2} [check] clsdmc_respget return: status=0, ecode=0
                          2013-05-17 01:13:16.918: [ora.gipcd][3481253632] {0:0:2} [check] clsdmc_respget return: status=0, ecode=0
                          2013-05-17 01:13:20.902: [    AGFW][3479152384] {0:0:118} Agent received the message: RESOURCE_START[ora.asm 1 1] ID 4098:4938
                          2013-05-17 01:13:20.902: [    AGFW][3479152384] {0:0:118} Preparing START command for: ora.asm 1 1
                          2013-05-17 01:13:20.902: [    AGFW][3479152384] {0:0:118} ora.asm 1 1 state changed from: OFFLINE to: STARTING
                          2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] (:CLSN00107:) clsn_agent::start {                                                  
                          2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start {                                                                 
                          2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection
                          2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 0
                          2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 0
                          2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection sid +ASM1 status  1                                 
                          2013-05-17 01:13:20.905: [ USRTHRD][3477051136] {0:0:118} ConnectionPool::~ConnectionPool m_oracleHome:/u01/app/11.2.0/grid, m_oracleSid:+ASM1, m_usrOraEnv:
                          2013-05-17 01:13:20.905: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::refresh
                          2013-05-17 01:13:20.905: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::refresh ORACLE_HOME = /u01/app/11.2.0/grid
                          2013-05-17 01:13:20.905: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 2 oracle_sid =
                          2013-05-17 01:13:20.905: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 4 oracle_sid =
                          2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] Node 0 doesn't exist
                          2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::getOracleSid: Exception IntException
                          2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::getOracleSidAttrib getCssNodeName Error: Agents: clssnsqlnum ret = 12:
                          2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 5 oracle_sid = +ASM1                                         
                          2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid oracle_sid = +ASM1                                           
                          2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::refresh ORACLE_SID = +ASM1                                               
                          2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] getOracleHomeAttrib: oracle_home = /u01/app/11.2.0/grid
                          2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/app/11.2.0/grid, m_oracleSid:+ASM1, m_usrOraEnv:
                          2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] updating GEN_USR_ORA_INST_NAME to +ASM1                                            
                          2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] clsnUtils::setResAttrib nonPerX current value GEN_USR_ORA_INST_NAME value +ASM1    
                          2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] setResAttrib clsagfw_modify_attribute attr GEN_USR_ORA_INST_NAME value +ASM1 retCode 0                                                                                                                                                  
                          2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] sModifyConfig for +ASM                                                             
                          2013-05-17 01:13:20.925: [ora.asm][3477051136] {0:0:118} [start] getOracleHomeAttrib: oracle_home = /u01/app/11.2.0/grid
                          2013-05-17 01:13:20.925: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
                          2013-05-17 01:13:20.925: [ora.asm][3477051136] {0:0:118} [start] sclsnInstAgent::sUpdateOratab oratab is /etc/oratab
                          2013-05-17 01:13:20.926: [ora.asm][3477051136] {0:0:118} [start] ConfigFile::setAltName altName = +ASM                                              
                          2013-05-17 01:13:20.926: [ora.asm][3477051136] {0:0:118} [start] ConfigFile::setAltValue altValue = /u01/app/11.2.0/grid:N
                          2013-05-17 01:13:20.926: [ora.asm][3477051136] {0:0:118} [start] sclsnInstAgent::sUpdateOratab CssLock
                          2013-05-17 01:13:20.930: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 2 oracle_sid =
                          2013-05-17 01:13:20.930: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 4 oracle_sid =
                          2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] Node 0 doesn't exist
                          2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::getOracleSid: Exception IntException
                          2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::getOracleSidAttrib getCssNodeName Error: Agents: clssnsqlnum ret = 12:
                          2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 5 oracle_sid = +ASM1                                         
                          2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid oracle_sid = +ASM1                                           
                          2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] ConfigFile::getAltEntry altEntry = +ASM1                                           
                          2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] ConfigFile::getValue name +asm1 value /u01/app/11.2.0/grid:N                       
                          2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] sclsnInstAgent::sUpdateOratab entry exists with dbName +ASM1 value /u01/app/11.2.0/grid:N                                                                                                                                               
                          2013-05-17 01:13:20.932: [ora.asm][3477051136] {0:0:118} [start] sclsnInstAgent::sUpdateOratab CSS unlock
                          2013-05-17 01:13:20.936: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
                          2013-05-17 01:13:20.936: [ora.asm][3477051136] {0:0:118} [start] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/11.2.0/grid Crshome /u01/app/11.2.0/grid
                          2013-05-17 01:13:20.936: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
                          2013-05-17 01:13:20.936: [ora.asm][3477051136] {0:0:118} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
                          2013-05-17 01:13:20.937: [ora.asm][3477051136] {0:0:118} [start] Container:start oracle home /u01/app/11.2.0/grid
                          2013-05-17 01:13:20.937: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt: server not attached
                          2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] ORA-01031: insufficient privileges

                          2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt (2) Exception OCIException
                          2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstConnection:connect:excp OCIException OCI error 1031
                          2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start: 1 errcode 1031
                          2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start: untrapped error
                          2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start: 2 errcode 1031
                          2013-05-17 01:13:20.963: [ USRTHRD][3477051136] {0:0:118} InstConnection:~InstConnection: this b0067eb0
                          2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] clean {                                               
                          2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop_option stop mode immediate option 1
                          2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop {                                     
                          2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop original reason system do shutdown abort
                          2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::resetConnection s_statusOfConnectionMap 00a87d90
                          2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::resetConnection sid +ASM1 status  2             
                          2013-05-17 01:13:20.965: [ora.asm][3477051136] {0:0:118} [start] Gimh::check OH /u01/app/11.2.0/grid SID +ASM1                   
                          2013-05-17 01:13:20.965: [ora.asm][3477051136] {0:0:118} [start] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
                          2013-05-17 01:13:20.965: [ora.asm][3477051136] {0:0:118} [start] (:CLSN00006:)AsmAgent::check failed gimh state 0
                          2013-05-17 01:13:20.965: [ora.asm][3477051136] {0:0:118} [start] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
                          2013-05-17 01:13:21.133: [ COMMCRS][3442890496]clsc_connect: (0x7f54b00992d0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

                          2013-05-17 01:13:21.133: [ora.asm][3477051136] {0:0:118} [start] checkCrsStat 2 CLSCRS_STAT ret: 184
                          2013-05-17 01:13:21.133: [ora.asm][3477051136] {0:0:118} [start] clsnUtils::error Exception type=2 string=

                          2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::checkCbk: Exception UserErrorException
                          2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start]
                          2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] (:CLSN00006:)InstAgent::check: return unplanned offline
                          2013-05-17 01:13:21.134: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_query_ctx rc=0
                          2013-05-17 01:13:21.134: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_inst_ctx rc=0
                          2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection
                          2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 0
                          2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 0
                          2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection sid +ASM1 status  1    
                          2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] InstAgent::check checkCounter 0 prev clsagfw_res_status 3 current clsagfw_res_status 1
                          2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop shutdown mode: 4
                          2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::getConnection 160 ERROR sid +ASM1 pool status 1                    
                          2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::getConnection 260 pConnxn 00000000
                          2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop pool pConnxn 00000000
                          2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
                          2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/11.2.0/grid Crshome /u01/app/11.2.0/grid
                          2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
                          2013-05-17 01:13:21.136: [ora.asm][3477051136] {0:0:118} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
                          2013-05-17 01:13:21.136: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop non pool pConnxn 1 b0041400
                          2013-05-17 01:13:21.136: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt: server not attached
                          2013-05-17 01:13:21.161: [ora.asm][3477051136] {0:0:118} [start] ORA-01034: ORACLE not available
                          ORA-27101: shared memory realm does not exist
                          Linux-x86_64 Error: 2: No such file or directory
                          Process ID: 0
                          Session ID: 0 Serial number: 0

                          2013-05-17 01:13:21.161: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt (2) Exception OCIException
                          2013-05-17 01:13:21.162: [ora.asm][3477051136] {0:0:118} [start] InstConnection:connect:excp OCIException OCI error 1034
                          2013-05-17 01:13:21.162: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop: connect1 errcode 1034
                          2013-05-17 01:13:21.162: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop: connect2 oracleHome /u01/app/11.2.0/grid oracleSid +ASM1
                          2013-05-17 01:13:21.162: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt: server not attached
                          2013-05-17 01:13:21.187: [ora.asm][3477051136] {0:0:118} [start] ORA-01031: insufficient privileges

                          2013-05-17 01:13:21.187: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt (2) Exception OCIException
                          2013-05-17 01:13:21.187: [ora.asm][3477051136] {0:0:118} [start] InstConnection:connect:excp OCIException OCI error 1031
                          2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop: connect2 errcode 1031
                          2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] clsnUtils::error Exception type=2 string=
                          ORA-01031: insufficient privileges

                          2013-05-17 01:13:21.188: [    AGFW][3477051136] {0:0:118} sending status msg [ORA-01031: insufficient privileges
                          ] for start for resource: ora.asm 1 1
                          2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::releaseConnection InstConnection b0041400
                          2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 1
                          2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection sid +ASM1, InstConnection b0041400
                          2013-05-17 01:13:21.188: [ USRTHRD][3477051136] {0:0:118} InstConnection::breakCall pConnxn:b0041400 DetachLock:00a82418 m_pSvcH:b00cce80
                          2013-05-17 01:13:21.188: [ USRTHRD][3477051136] {0:0:118} InstConnection:~InstConnection: this b0041400
                          2013-05-17 01:13:21.189: [    AGFW][3479152384] {0:0:118} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:4938
                          2013-05-17 01:13:21.189: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection delete InstConnection b0041400
                          2013-05-17 01:13:21.189: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 1
                          2013-05-17 01:13:21.189: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection
                          2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 0
                          2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 0
                          2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection sid +ASM1 status  1                      
                          2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop db/asm
                          2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection
                          2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 0
                          2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 0
                          2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection sid +ASM1 status  1                      
                          2013-05-17 01:13:21.191: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop Container->stop, isShutdown = 0
                          2013-05-17 01:13:21.191: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop: }
                          2013-05-17 01:13:21.191: [ora.asm][3477051136] {0:0:118} [start] clean }
                          2013-05-17 01:13:21.191: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start exception }
                          2013-05-17 01:13:21.191: [   AGENT][3477051136] {0:0:118} UserErrorException: Locale is
                          2013-05-17 01:13:21.192: [ora.asm][3477051136] {0:0:118} [start] clsnUtils::error Exception type=2 string=
                          CRS-5017: The resource action "ora.asm start" encountered the following error:
                          ORA-01031: insufficient privileges
                          . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".

                          2013-05-17 01:13:21.192: [    AGFW][3477051136] {0:0:118} sending status msg [CRS-5017: The resource action "ora.asm start" encountered the following error:                                                                                                                                             
                          ORA-01031: insufficient privileges                                                                                                                  
                          . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".               
                          ] for start for resource: ora.asm 1 1
                          2013-05-17 01:13:21.192: [ora.asm][3477051136] {0:0:118} [start] (:CLSN00107:) clsn_agent::start }
                          2013-05-17 01:13:21.192: [    AGFW][3477051136] {0:0:118} Command: start for resource: ora.asm 1 1 completed with status: FAIL
                          2013-05-17 01:13:21.193: [    AGFW][3479152384] {0:0:118} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:4938
                          2013-05-17 01:13:21.193: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::resetConnection s_statusOfConnectionMap 00a87d90
                          2013-05-17 01:13:21.193: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::resetConnection sid +ASM1 status  2                                
                          2013-05-17 01:13:21.193: [ora.asm][3477051136] {0:0:118} [check] Gimh::check OH /u01/app/11.2.0/grid SID +ASM1                                      
                          2013-05-17 01:13:21.194: [    AGFW][3479152384] {0:0:118} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:4938
                          2013-05-17 01:13:21.194: [ora.asm][3477051136] {0:0:118} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
                          2013-05-17 01:13:21.194: [ora.asm][3477051136] {0:0:118} [check] (:CLSN00006:)AsmAgent::check failed gimh state 0
                          2013-05-17 01:13:21.194: [ora.asm][3477051136] {0:0:118} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
                          2013-05-17 01:13:21.362: [ COMMCRS][3442890496]clsc_connect: (0x7f54b00992d0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

                          2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check] checkCrsStat 2 CLSCRS_STAT ret: 184
                          2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check] clsnUtils::error Exception type=2 string=

                          2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check] AsmAgent::checkCbk: Exception UserErrorException
                          2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check]
                          2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check] (:CLSN00006:)InstAgent::check: return unplanned offline
                          2013-05-17 01:13:21.363: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_query_ctx rc=0
                          2013-05-17 01:13:21.363: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_inst_ctx rc=0
                          2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::stopConnection
                          2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::removeConnection connection count 0
                          2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::removeConnection freed 0
                          2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::stopConnection sid +ASM1 status  1    
                          2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] InstAgent::check checkCounter 0 prev clsagfw_res_status 1 current clsagfw_res_status 1
                          2013-05-17 01:13:21.364: [    AGFW][3479152384] {0:0:118} ora.asm 1 1 state changed from: STARTING to: OFFLINE
                          2013-05-17 01:13:21.365: [    AGFW][3479152384] {0:0:118} Agent sending last reply for: RESOURCE_START[ora.asm 1 1] ID 4098:4938
                          2013-05-17 01:13:21.373: [    AGFW][3479152384] {0:0:118} Agent received the message: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:4944
                          2013-05-17 01:13:21.373: [    AGFW][3479152384] {0:0:118} Preparing CLEAN command for: ora.asm 1 1
                          2013-05-17 01:13:21.373: [    AGFW][3479152384] {0:0:118} ora.asm 1 1 state changed from: OFFLINE to: CLEANING
                          2013-05-17 01:13:21.374: [ora.asm][3477051136] {0:0:118} [clean] (:CLSN00106:) clsn_agent::clean {                                                  
                          2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] clean {                                                                            
                          2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop_option stop mode immediate option 1
                          2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop {                                                                  
                          2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop original reason system do shutdown abort
                          2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::resetConnection s_statusOfConnectionMap 00a87d90
                          2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::resetConnection sid +ASM1 status  2                                
                          2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] Gimh::check OH /u01/app/11.2.0/grid SID +ASM1                                      
                          2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
                          2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] (:CLSN00006:)AsmAgent::check failed gimh state 0
                          2013-05-17 01:13:21.376: [ora.asm][3477051136] {0:0:118} [clean] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
                          2013-05-17 01:13:21.542: [ COMMCRS][3442890496]clsc_connect: (0x7f54b00993b0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

                          2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] checkCrsStat 2 CLSCRS_STAT ret: 184
                          2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] clsnUtils::error Exception type=2 string=

                          2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] AsmAgent::checkCbk: Exception UserErrorException
                          2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean]
                          2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] (:CLSN00006:)InstAgent::check: return unplanned offline
                          2013-05-17 01:13:21.543: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_query_ctx rc=0
                          2013-05-17 01:13:21.543: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_inst_ctx rc=0
                          2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::stopConnection
                          2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::removeConnection connection count 0
                          2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::removeConnection freed 0
                          2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::stopConnection sid +ASM1 status  1    
                          2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::check checkCounter 0 prev clsagfw_res_status 1 current clsagfw_res_status 1
                          2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop shutdown mode: 4
                          2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::getConnection 160 ERROR sid +ASM1 pool status 1                    
                          2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::getConnection 260 pConnxn 00000000
                          2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop pool pConnxn 00000000
                          2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
                          2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/11.2.0/grid Crshome /u01/app/11.2.0/grid
                          2013-05-17 01:13:21.544: [ora                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                          • 10. Re: unable to start asm on one of the nodes
                            Red8
                            here is the out put that i have

                            [root@KIFIYA-UBS-ORA-RAC1 bin]# ./crsctl check crs
                            CRS-4638: Oracle High Availability Services is online
                            CRS-4535: Cannot communicate with Cluster Ready Services
                            CRS-4529: Cluster Synchronization Services is online
                            CRS-4534: Cannot communicate with Event Manager
                            [root@KIFIYA-UBS-ORA-RAC1 bin]# ./crsctl stat res -t -init
                            --------------------------------------------------------------------------------
                            NAME TARGET STATE SERVER STATE_DETAILS
                            --------------------------------------------------------------------------------
                            Cluster Resources
                            --------------------------------------------------------------------------------
                            ora.asm
                            1 ONLINE OFFLINE
                            ora.cluster_interconnect.haip
                            1 ONLINE ONLINE kifiya-ubs-ora-rac1
                            ora.crf
                            1 ONLINE ONLINE kifiya-ubs-ora-rac1
                            ora.crsd
                            1 ONLINE OFFLINE
                            ora.cssd
                            1 ONLINE ONLINE kifiya-ubs-ora-rac1
                            ora.cssdmonitor
                            1 ONLINE ONLINE kifiya-ubs-ora-rac1
                            ora.ctssd
                            1 ONLINE ONLINE kifiya-ubs-ora-rac1 ACTIVE:0
                            ora.diskmon
                            1 OFFLINE OFFLINE
                            ora.evmd
                            1 ONLINE INTERMEDIATE kifiya-ubs-ora-rac1
                            ora.gipcd
                            1 ONLINE ONLINE kifiya-ubs-ora-rac1
                            ora.gpnpd
                            1 ONLINE ONLINE kifiya-ubs-ora-rac1
                            ora.mdnsd
                            1 ONLINE ONLINE kifiya-ubs-ora-rac1
                            [root@KIFIYA-UBS-ORA-RAC1 bin]#
                            • 11. Re: unable to start asm on one of the nodes
                              rarain
                              As I seen in your log all devices have root privilege which is probably preventing Grid user to accessing physical disk and not able to start ASM instance which in turn make OCR devices inaccessible. Suggesting you to please assign appropriate privilege on the physical disks and retry to start CRS.

                              [root@KIFIYA-UBS-ORA-RAC1 bin]# ls -l /dev/mapper/
                              total 0
                              crw-rw---- 1 root root 10, 58 May 17 10:16 control
                              lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_DATA1 -> ../dm-2
                              lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_DATA2 -> ../dm-8
                              lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_FRA1 -> ../dm-7
                              lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_FRA2 -> ../dm-6
                              lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK1 -> ../dm-0
                              lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK2 -> ../dm-3
                              lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK3 -> ../dm-5
                              lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK4 -> ../dm-4
                              lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK5 -> ../dm-1

                              Thanks

                              Edited by: rarain on May 17, 2013 11:20 AM
                              • 12. Re: unable to start asm on one of the nodes
                                Santysharma-Oracle
                                Hi,

                                Please check your grid user has right group privileges, you can compare with other nodes in cluster.

                                Also try to manually login to ASM as grid user, most likely toy will get same error.



                                Regards,
                                Sharma
                                • 13. Re: unable to start asm on one of the nodes
                                  Red8
                                  I have fixed it and here is how i done it. Actually it is very easy but it took 2 days for me to figure it out and the problem was my fello DB admin has changed
                                  the sqlnet.ora file authnetication to none. Because of this the ASM instance had a trouble bring it up.

                                  So what i did was, i commented out the autentication from the two sqlnet.ora files and it restarted the server, And everything worked perfectly

                                  Thanks for all your fox u tried to help, i greatly apperciated it.

                                  Thanks
                                  Rediet