This discussion is archived
13 Replies Latest reply: May 24, 2013 2:12 AM by Red8 RSS

unable to start asm on one of the nodes

Red8 Newbie
Currently Being Moderated
Dear Friends

due to some problem with our AC all servers in our datacenter where powered off, and after all is fixed and try to start the RAC, it is showing the following errors,
can you please help me


[root@KIFIYA-UBS-ORA-RAC1 bin]# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.asm' on 'kifiya-ubs-ora-rac1'
ORA-01031: insufficient privileges
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-01031: insufficient privileges
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".
CRS-2674: Start of 'ora.asm' on 'kifiya-ubs-ora-rac1' failed
CRS-2679: Attempting to clean 'ora.asm' on 'kifiya-ubs-ora-rac1'
ORA-01031: insufficient privileges
CRS-2681: Clean of 'ora.asm' on 'kifiya-ubs-ora-rac1' succeeded
CRS-4000: Command Start failed, or completed with errors.
  • 1. Re: unable to start asm on one of the nodes
    AshishShukla Newbie
    Currently Being Moderated
    Hello,

    I guess you are using 11gR2 RAC in your environment. So the problem is related with kernel parameters *"/etc/sysctl.conf"*

    Actually sometimes if we are using *"RunFixup"* at the time of installation to modify kernel parameters then this kind of problems occurs.

    So finally, solution is modify kernel parameter file on that machine manually. Hope it will work.


    Thanks,
    Ashish Shukla
  • 2. Re: unable to start asm on one of the nodes
    Red8 Newbie
    Currently Being Moderated
    thanks for your response,

    yes i am using 11gR2.0.3 RAC with red hat 6.3 and it has been working for the last 3 month, all was ok, installation went smoothly

    but know after the restart this node is creating problems,

    kernel parameters are fine also, so what could be the problem ?


    here is the trace file output, may be Unable to get log name be the problem cos most of the trace file is filled with this

    2013-05-05 03:04:31.067: [ default]failed to initialize skgp context
    2013-05-05 03:04:31.067: [ default]slos op : sslssreghdlr
    2013-05-05 03:04:31.067: [ default]slos dep : Error 0 (0)
    2013-05-05 03:04:31.067: [ default]slos loc : sskgpinit1
    2013-05-05 03:04:31.067: [ default]slos info:
    [   CLWAL]clsw_Initialize: OLR initlevel [30000]
    2013-05-05 03:04:31.069: [ default]a_init: Unable to get log name. Retval:[-4]
    2013-05-05 03:04:31.096: [    GPNP]clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:313] Result: (0) CLSGPNP_OK. got ASM-Profile.DiscoveryString='/dev/mapper/*'
    kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

    *** 2013-05-16 08:59:48.294
    kfgbUnbind: unbinding group 2/0xab2a4083 (FRA_DATA)
    kfgbUnregister: unregistering group 2/0xab2a4083 (FRA_DATA)

    *** 2013-05-16 08:59:58.305
    kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
    kfdp_query(DATA): 23
    ----- Abridged Call Stack Trace -----
    ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
    <-__libc_start_main()+253<-_start()+36                                                                                                              
    ----- End of Abridged Call Stack Trace -----

    *** 2013-05-16 09:00:08.690
    kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

    *** 2013-05-16 09:00:18.701
    kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
    kfdp_query(DATA): 24
    ----- Abridged Call Stack Trace -----
    ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
    <-__libc_start_main()+253<-_start()+36                                                                                                              
    ----- End of Abridged Call Stack Trace -----

    *** 2013-05-16 09:00:28.901
    kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

    *** 2013-05-16 09:00:38.913
    kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
    kfdp_query(DATA): 25
    ----- Abridged Call Stack Trace -----
    ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
    <-__libc_start_main()+253<-_start()+36
    ----- End of Abridged Call Stack Trace -----

    *** 2013-05-16 09:00:49.109
    kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

    *** 2013-05-16 09:00:59.120
    kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
    kfdp_query(DATA): 26
    ----- Abridged Call Stack Trace -----
    ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
    <-__libc_start_main()+253<-_start()+36
    ----- End of Abridged Call Stack Trace -----

    *** 2013-05-16 09:01:09.353
    kfgbUnbind: unbinding group 1/0x379a4064 (DATA)

    *** 2013-05-16 09:01:19.363
    kfgbBind: binding kfgpn for group 1/0x379A4064 (DATA)
    kfdp_query(DATA): 27
    ----- Abridged Call Stack Trace -----
    ksedsts()+461<-kfdp_query()+530<-kfiRefresh()+136<-kfi_set_cb()+2149<-kfeInvokeLockCallback()+2733<-kfgbAttrLckRegister()+309<-kfgbDriver()+2193<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201
    <-__libc_start_main()+253<-_start()+36
    ----- End of Abridged Call Stack Trace -----

    Edited by: Red8 on May 16, 2013 9:34 PM
  • 3. Re: unable to start asm on one of the nodes
    AshishShukla Newbie
    Currently Being Moderated
    Yes, it happens when you will restart your machines kernel parameters start misbehaving if you have updated kernel parameter with OUI runfixup.

    Modify kernel parameters again manually.

    fs.suid_dumpable = 1
    fs.aio-max-nr = 1048576
    fs.file-max = 6815744
    kernel.shmall = 2097152
    kernel.shmmax = 536870912
    kernel.shmmni = 4096
    kernel.sem = 250 32000 100 128
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default = 262144
    net.core.rmem_max = 4194304
    net.core.wmem_default = 262144
    net.core.wmem_max = 1048586
  • 4. Re: unable to start asm on one of the nodes
    Red8 Newbie
    Currently Being Moderated
    Thanks for your help, i really apperciate it

    This are what i have in my /etc/sysctl.conf.

    fs.file-max = 6815744
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default = 262144
    net.core.rmem_max = 4194304
    net.core.wmem_default = 262144
    net.core.wmem_max = 1048576
    kernel.shmall = 8189848
    kernel.shmmax = 33545619456
    kernel.shmmni = 4096
    kernel.sem = 250 32000 100 128
  • 5. Re: unable to start asm on one of the nodes
    AshishShukla Newbie
    Currently Being Moderated
    Optionally do can do one more thing, Copy "/etc/sysctl.conf" file from the node where ASM is running and paste to the problematic machine.

    # systcl -p

    and then try to start ASM again. If still problem is there then please share "Clusterware Alert log file" and "crsd.log" files.


    Thanks,
    Ashish Shukla
  • 6. Re: unable to start asm on one of the nodes
    Red8 Newbie
    Currently Being Moderated
    I have copied the /etc/sysctl.conf file from the working node to the failed node ad tried to restart the asm instance, but it is not working.

    so i please take a look at the logs

    1.Clusterware Alert log file

    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
    2013-05-16 23:58:44.864
    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
    2013-05-17 00:29:33.443
    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
    2013-05-17 00:59:57.993
    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
    2013-05-17 01:13:20.965
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 01:13:21.194
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 01:13:21.375
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 01:13:21.601
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 01:31:10.556
    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
    2013-05-17 01:36:06.063
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 01:36:06.284
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 01:36:06.466
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 01:36:06.688
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 02:01:59.110
    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
    2013-05-17 02:15:16.234
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 02:15:16.462
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 02:15:16.643
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 02:15:16.862
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 02:29:11.423
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 02:29:11.651
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 02:29:11.833
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 02:29:12.055
    [u01/app/11.2.0/grid/bin/oraagent.bin(5557)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log"
    2013-05-17 02:32:07.671
    [ctssd(5798)]CRS-2408:The clock on host kifiya-ubs-ora-rac1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
  • 7. Re: unable to start asm on one of the nodes
    Red8 Newbie
    Currently Being Moderated
    2. crsd.log

    [root@KIFIYA-UBS-ORA-RAC1 bin]# cat /u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/crsd/crsd.log

    2013-05-17 10:53:01.713: [ default][2997434144] First attempt: init CSS context succeeded.
    2013-05-17 10:53:01.713: [ default][2997434144] CRS Daemon Starting
    2013-05-17 10:53:01.714: [ CRSMAIN][2997434144] Checking the OCR device
    2013-05-17 10:53:01.715: [ CRSMAIN][2997434144] Sync-up with OCR
    2013-05-17 10:53:01.715: [ CRSMAIN][2997434144] Connecting to the CSS Daemon
    2013-05-17 10:53:01.715: [ CRSMAIN][2997434144] Getting local node number
    2013-05-17 10:53:01.715: [ CRSMAIN][2997434144] Initializing OCR
    [  clsdmt][2982430464]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=kifiya-ubs-ora-rac1DBG_CRSD))
    2013-05-17 10:53:01.717: [  clsdmt][2982430464]PID for the Process [14363], connkey 1
    2013-05-17 10:53:01.717: [  clsdmt][2982430464]Creating PID [14363] file for home /u01/app/11.2.0/grid host kifiya-ubs-ora-rac1 bin crs to /u01/app/11.2.0/grid/crs/init/
    2013-05-17 10:53:01.717: [  clsdmt][2982430464]Writing PID [14363] to the file [u01/app/11.2.0/grid/crs/init/kifiya-ubs-ora-rac1.pid]
    [   CLWAL][2997434144]clsw_Initialize: OLR initlevel [70000]
    2013-05-17 10:53:04.012: [  OCRASM][2997434144]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][2997434144]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

    2013-05-17 10:53:04.012: [  OCRASM][2997434144]ASM Error Stack : ORA-15077: could not locate ASM instance serving a required diskgroup

    2013-05-17 10:53:04.015: [  OCRASM][2997434144]proprasmo: kgfoCheckMount returned [7]
    2013-05-17 10:53:04.015: [  OCRASM][2997434144]proprasmo: The ASM instance is down
    2013-05-17 10:53:04.015: [  OCRRAW][2997434144]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2013-05-17 10:53:04.015: [  OCRRAW][2997434144]proprioo: No OCR/OLR devices are usable
    2013-05-17 10:53:04.015: [  OCRASM][2997434144]proprasmcl: asmhandle is NULL
    2013-05-17 10:53:04.016: [    GIPC][2997434144] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 690], original from [clsss.c : 5326]
    2013-05-17 10:53:04.017: [ default][2997434144]clsvactversion:4: Retrieving Active Version from local storage.
    2013-05-17 10:53:04.020: [  OCRRAW][2997434144]proprrepauto: The local OCR configuration matches with the configuration published by OCR Cache Writer. No repair required.
    2013-05-17 10:53:04.022: [  OCRRAW][2997434144]proprinit: Could not open raw device
    2013-05-17 10:53:04.022: [  OCRASM][2997434144]proprasmcl: asmhandle is NULL
    2013-05-17 10:53:04.024: [  OCRAPI][2997434144]a_init:16!: Backend init unsuccessful : [26]
    2013-05-17 10:53:04.025: [  CRSOCR][2997434144] OCR context init failure. Error: PROC-26: Error while accessing the physical storage
    ORA-15077: could not locate ASM instance serving a required diskgroup

    2013-05-17 10:53:04.025: [ CRSMAIN][2997434144] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-26: Error while accessing the physical storage
    ORA-15077: could not locate ASM instance serving a required diskgroup

    2013-05-17 10:53:04.025: [    CRSD][2997434144][PANIC] CRSD exiting: Could not init OCR, code: 26
    2013-05-17 10:53:04.025: [    CRSD][2997434144] Done.

    [root@KIFIYA-UBS-ORA-RAC1 bin]#
    [root@KIFIYA-UBS-ORA-RAC1 bin]#
    [root@KIFIYA-UBS-ORA-RAC1 bin]# ls -l /dev/mapper/
    total 0
    crw-rw---- 1 root root 10, 58 May 17 10:16 control
    lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_DATA1 -> ../dm-2
    lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_DATA2 -> ../dm-8
    lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_FRA1 -> ../dm-7
    lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_FRA2 -> ../dm-6
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK1 -> ../dm-0
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK2 -> ../dm-3
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK3 -> ../dm-5
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK4 -> ../dm-4
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK5 -> ../dm-1

    [root@KIFIYA-UBS-ORA-RAC1 bin]# ./crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    -- ----- ----------------- --------- ---------
    1. ONLINE 980d4cd4601e4f99bfa880f99a2ba34b (/dev/mapper/VOTE_DSK5) [DATA]
    2. ONLINE 7dc793a3ba6b4f5dbf7387f6644f46da (/dev/mapper/VOTE_DSK4) [DATA]
    3. ONLINE 9931f77f96044f15bf779f8ebe0eb028 (/dev/mapper/VOTE_DSK3) [DATA]
    4. ONLINE a1870854f20e4f49bf514541b9af9d29 (/dev/mapper/VOTE_DSK2) [DATA]
    5. ONLINE 15999290b89f4f89bf0b27edc9079ec7 (/dev/mapper/VOTE_DSK1) [DATA]
    Located 5 voting disk(s).
    [root@KIFIYA-UBS-ORA-RAC1 bin]

    Edited by: Red8 on May 17, 2013 1:49 AM
  • 8. Re: unable to start asm on one of the nodes
    AshishShukla Newbie
    Currently Being Moderated
    please share this log file:

    "*/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".*
  • 9. Re: unable to start asm on one of the nodes
    Red8 Newbie
    Currently Being Moderated
    2013-05-17 01:13:13.720: [ora.mdnsd][3477051136] {0:0:2} [check] clsdmc_respget return: status=0, ecode=0
    2013-05-17 01:13:15.345: [ora.gpnpd][3477051136] {0:0:2} [check] clsdmc_respget return: status=0, ecode=0
    2013-05-17 01:13:16.918: [ora.gipcd][3481253632] {0:0:2} [check] clsdmc_respget return: status=0, ecode=0
    2013-05-17 01:13:20.902: [    AGFW][3479152384] {0:0:118} Agent received the message: RESOURCE_START[ora.asm 1 1] ID 4098:4938
    2013-05-17 01:13:20.902: [    AGFW][3479152384] {0:0:118} Preparing START command for: ora.asm 1 1
    2013-05-17 01:13:20.902: [    AGFW][3479152384] {0:0:118} ora.asm 1 1 state changed from: OFFLINE to: STARTING
    2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] (:CLSN00107:) clsn_agent::start {                                                  
    2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start {                                                                 
    2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection
    2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 0
    2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 0
    2013-05-17 01:13:20.904: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection sid +ASM1 status  1                                 
    2013-05-17 01:13:20.905: [ USRTHRD][3477051136] {0:0:118} ConnectionPool::~ConnectionPool m_oracleHome:/u01/app/11.2.0/grid, m_oracleSid:+ASM1, m_usrOraEnv:
    2013-05-17 01:13:20.905: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::refresh
    2013-05-17 01:13:20.905: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::refresh ORACLE_HOME = /u01/app/11.2.0/grid
    2013-05-17 01:13:20.905: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 2 oracle_sid =
    2013-05-17 01:13:20.905: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 4 oracle_sid =
    2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] Node 0 doesn't exist
    2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::getOracleSid: Exception IntException
    2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::getOracleSidAttrib getCssNodeName Error: Agents: clssnsqlnum ret = 12:
    2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 5 oracle_sid = +ASM1                                         
    2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid oracle_sid = +ASM1                                           
    2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::refresh ORACLE_SID = +ASM1                                               
    2013-05-17 01:13:20.919: [ora.asm][3477051136] {0:0:118} [start] getOracleHomeAttrib: oracle_home = /u01/app/11.2.0/grid
    2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/app/11.2.0/grid, m_oracleSid:+ASM1, m_usrOraEnv:
    2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] updating GEN_USR_ORA_INST_NAME to +ASM1                                            
    2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] clsnUtils::setResAttrib nonPerX current value GEN_USR_ORA_INST_NAME value +ASM1    
    2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] setResAttrib clsagfw_modify_attribute attr GEN_USR_ORA_INST_NAME value +ASM1 retCode 0                                                                                                                                                  
    2013-05-17 01:13:20.920: [ora.asm][3477051136] {0:0:118} [start] sModifyConfig for +ASM                                                             
    2013-05-17 01:13:20.925: [ora.asm][3477051136] {0:0:118} [start] getOracleHomeAttrib: oracle_home = /u01/app/11.2.0/grid
    2013-05-17 01:13:20.925: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
    2013-05-17 01:13:20.925: [ora.asm][3477051136] {0:0:118} [start] sclsnInstAgent::sUpdateOratab oratab is /etc/oratab
    2013-05-17 01:13:20.926: [ora.asm][3477051136] {0:0:118} [start] ConfigFile::setAltName altName = +ASM                                              
    2013-05-17 01:13:20.926: [ora.asm][3477051136] {0:0:118} [start] ConfigFile::setAltValue altValue = /u01/app/11.2.0/grid:N
    2013-05-17 01:13:20.926: [ora.asm][3477051136] {0:0:118} [start] sclsnInstAgent::sUpdateOratab CssLock
    2013-05-17 01:13:20.930: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 2 oracle_sid =
    2013-05-17 01:13:20.930: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 4 oracle_sid =
    2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] Node 0 doesn't exist
    2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::getOracleSid: Exception IntException
    2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::getOracleSidAttrib getCssNodeName Error: Agents: clssnsqlnum ret = 12:
    2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid 5 oracle_sid = +ASM1                                         
    2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] AsmAgent:getOracleSid oracle_sid = +ASM1                                           
    2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] ConfigFile::getAltEntry altEntry = +ASM1                                           
    2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] ConfigFile::getValue name +asm1 value /u01/app/11.2.0/grid:N                       
    2013-05-17 01:13:20.931: [ora.asm][3477051136] {0:0:118} [start] sclsnInstAgent::sUpdateOratab entry exists with dbName +ASM1 value /u01/app/11.2.0/grid:N                                                                                                                                               
    2013-05-17 01:13:20.932: [ora.asm][3477051136] {0:0:118} [start] sclsnInstAgent::sUpdateOratab CSS unlock
    2013-05-17 01:13:20.936: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
    2013-05-17 01:13:20.936: [ora.asm][3477051136] {0:0:118} [start] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/11.2.0/grid Crshome /u01/app/11.2.0/grid
    2013-05-17 01:13:20.936: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
    2013-05-17 01:13:20.936: [ora.asm][3477051136] {0:0:118} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
    2013-05-17 01:13:20.937: [ora.asm][3477051136] {0:0:118} [start] Container:start oracle home /u01/app/11.2.0/grid
    2013-05-17 01:13:20.937: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt: server not attached
    2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] ORA-01031: insufficient privileges

    2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt (2) Exception OCIException
    2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstConnection:connect:excp OCIException OCI error 1031
    2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start: 1 errcode 1031
    2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start: untrapped error
    2013-05-17 01:13:20.963: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start: 2 errcode 1031
    2013-05-17 01:13:20.963: [ USRTHRD][3477051136] {0:0:118} InstConnection:~InstConnection: this b0067eb0
    2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] clean {                                               
    2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop_option stop mode immediate option 1
    2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop {                                     
    2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop original reason system do shutdown abort
    2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::resetConnection s_statusOfConnectionMap 00a87d90
    2013-05-17 01:13:20.964: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::resetConnection sid +ASM1 status  2             
    2013-05-17 01:13:20.965: [ora.asm][3477051136] {0:0:118} [start] Gimh::check OH /u01/app/11.2.0/grid SID +ASM1                   
    2013-05-17 01:13:20.965: [ora.asm][3477051136] {0:0:118} [start] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
    2013-05-17 01:13:20.965: [ora.asm][3477051136] {0:0:118} [start] (:CLSN00006:)AsmAgent::check failed gimh state 0
    2013-05-17 01:13:20.965: [ora.asm][3477051136] {0:0:118} [start] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
    2013-05-17 01:13:21.133: [ COMMCRS][3442890496]clsc_connect: (0x7f54b00992d0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

    2013-05-17 01:13:21.133: [ora.asm][3477051136] {0:0:118} [start] checkCrsStat 2 CLSCRS_STAT ret: 184
    2013-05-17 01:13:21.133: [ora.asm][3477051136] {0:0:118} [start] clsnUtils::error Exception type=2 string=

    2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] AsmAgent::checkCbk: Exception UserErrorException
    2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start]
    2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] (:CLSN00006:)InstAgent::check: return unplanned offline
    2013-05-17 01:13:21.134: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_query_ctx rc=0
    2013-05-17 01:13:21.134: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_inst_ctx rc=0
    2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection
    2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 0
    2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 0
    2013-05-17 01:13:21.134: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection sid +ASM1 status  1    
    2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] InstAgent::check checkCounter 0 prev clsagfw_res_status 3 current clsagfw_res_status 1
    2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop shutdown mode: 4
    2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::getConnection 160 ERROR sid +ASM1 pool status 1                    
    2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::getConnection 260 pConnxn 00000000
    2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop pool pConnxn 00000000
    2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
    2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/11.2.0/grid Crshome /u01/app/11.2.0/grid
    2013-05-17 01:13:21.135: [ora.asm][3477051136] {0:0:118} [start] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
    2013-05-17 01:13:21.136: [ora.asm][3477051136] {0:0:118} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/11.2.0/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
    2013-05-17 01:13:21.136: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop non pool pConnxn 1 b0041400
    2013-05-17 01:13:21.136: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt: server not attached
    2013-05-17 01:13:21.161: [ora.asm][3477051136] {0:0:118} [start] ORA-01034: ORACLE not available
    ORA-27101: shared memory realm does not exist
    Linux-x86_64 Error: 2: No such file or directory
    Process ID: 0
    Session ID: 0 Serial number: 0

    2013-05-17 01:13:21.161: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt (2) Exception OCIException
    2013-05-17 01:13:21.162: [ora.asm][3477051136] {0:0:118} [start] InstConnection:connect:excp OCIException OCI error 1034
    2013-05-17 01:13:21.162: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop: connect1 errcode 1034
    2013-05-17 01:13:21.162: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop: connect2 oracleHome /u01/app/11.2.0/grid oracleSid +ASM1
    2013-05-17 01:13:21.162: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt: server not attached
    2013-05-17 01:13:21.187: [ora.asm][3477051136] {0:0:118} [start] ORA-01031: insufficient privileges

    2013-05-17 01:13:21.187: [ora.asm][3477051136] {0:0:118} [start] InstConnection::connectInt (2) Exception OCIException
    2013-05-17 01:13:21.187: [ora.asm][3477051136] {0:0:118} [start] InstConnection:connect:excp OCIException OCI error 1031
    2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop: connect2 errcode 1031
    2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] clsnUtils::error Exception type=2 string=
    ORA-01031: insufficient privileges

    2013-05-17 01:13:21.188: [    AGFW][3477051136] {0:0:118} sending status msg [ORA-01031: insufficient privileges
    ] for start for resource: ora.asm 1 1
    2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::releaseConnection InstConnection b0041400
    2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 1
    2013-05-17 01:13:21.188: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection sid +ASM1, InstConnection b0041400
    2013-05-17 01:13:21.188: [ USRTHRD][3477051136] {0:0:118} InstConnection::breakCall pConnxn:b0041400 DetachLock:00a82418 m_pSvcH:b00cce80
    2013-05-17 01:13:21.188: [ USRTHRD][3477051136] {0:0:118} InstConnection:~InstConnection: this b0041400
    2013-05-17 01:13:21.189: [    AGFW][3479152384] {0:0:118} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:4938
    2013-05-17 01:13:21.189: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection delete InstConnection b0041400
    2013-05-17 01:13:21.189: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 1
    2013-05-17 01:13:21.189: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection
    2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 0
    2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 0
    2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection sid +ASM1 status  1                      
    2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop db/asm
    2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection
    2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection connection count 0
    2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::removeConnection freed 0
    2013-05-17 01:13:21.190: [ora.asm][3477051136] {0:0:118} [start] ConnectionPool::stopConnection sid +ASM1 status  1                      
    2013-05-17 01:13:21.191: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop Container->stop, isShutdown = 0
    2013-05-17 01:13:21.191: [ora.asm][3477051136] {0:0:118} [start] InstAgent::stop: }
    2013-05-17 01:13:21.191: [ora.asm][3477051136] {0:0:118} [start] clean }
    2013-05-17 01:13:21.191: [ora.asm][3477051136] {0:0:118} [start] InstAgent::start exception }
    2013-05-17 01:13:21.191: [   AGENT][3477051136] {0:0:118} UserErrorException: Locale is
    2013-05-17 01:13:21.192: [ora.asm][3477051136] {0:0:118} [start] clsnUtils::error Exception type=2 string=
    CRS-5017: The resource action "ora.asm start" encountered the following error:
    ORA-01031: insufficient privileges
    . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".

    2013-05-17 01:13:21.192: [    AGFW][3477051136] {0:0:118} sending status msg [CRS-5017: The resource action "ora.asm start" encountered the following error:                                                                                                                                             
    ORA-01031: insufficient privileges                                                                                                                  
    . For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/kifiya-ubs-ora-rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".               
    ] for start for resource: ora.asm 1 1
    2013-05-17 01:13:21.192: [ora.asm][3477051136] {0:0:118} [start] (:CLSN00107:) clsn_agent::start }
    2013-05-17 01:13:21.192: [    AGFW][3477051136] {0:0:118} Command: start for resource: ora.asm 1 1 completed with status: FAIL
    2013-05-17 01:13:21.193: [    AGFW][3479152384] {0:0:118} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:4938
    2013-05-17 01:13:21.193: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::resetConnection s_statusOfConnectionMap 00a87d90
    2013-05-17 01:13:21.193: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::resetConnection sid +ASM1 status  2                                
    2013-05-17 01:13:21.193: [ora.asm][3477051136] {0:0:118} [check] Gimh::check OH /u01/app/11.2.0/grid SID +ASM1                                      
    2013-05-17 01:13:21.194: [    AGFW][3479152384] {0:0:118} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:4938
    2013-05-17 01:13:21.194: [ora.asm][3477051136] {0:0:118} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
    2013-05-17 01:13:21.194: [ora.asm][3477051136] {0:0:118} [check] (:CLSN00006:)AsmAgent::check failed gimh state 0
    2013-05-17 01:13:21.194: [ora.asm][3477051136] {0:0:118} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
    2013-05-17 01:13:21.362: [ COMMCRS][3442890496]clsc_connect: (0x7f54b00992d0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

    2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check] checkCrsStat 2 CLSCRS_STAT ret: 184
    2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check] clsnUtils::error Exception type=2 string=

    2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check] AsmAgent::checkCbk: Exception UserErrorException
    2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check]
    2013-05-17 01:13:21.363: [ora.asm][3477051136] {0:0:118} [check] (:CLSN00006:)InstAgent::check: return unplanned offline
    2013-05-17 01:13:21.363: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_query_ctx rc=0
    2013-05-17 01:13:21.363: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_inst_ctx rc=0
    2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::stopConnection
    2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::removeConnection connection count 0
    2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::removeConnection freed 0
    2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] ConnectionPool::stopConnection sid +ASM1 status  1    
    2013-05-17 01:13:21.364: [ora.asm][3477051136] {0:0:118} [check] InstAgent::check checkCounter 0 prev clsagfw_res_status 1 current clsagfw_res_status 1
    2013-05-17 01:13:21.364: [    AGFW][3479152384] {0:0:118} ora.asm 1 1 state changed from: STARTING to: OFFLINE
    2013-05-17 01:13:21.365: [    AGFW][3479152384] {0:0:118} Agent sending last reply for: RESOURCE_START[ora.asm 1 1] ID 4098:4938
    2013-05-17 01:13:21.373: [    AGFW][3479152384] {0:0:118} Agent received the message: RESOURCE_CLEAN[ora.asm 1 1] ID 4100:4944
    2013-05-17 01:13:21.373: [    AGFW][3479152384] {0:0:118} Preparing CLEAN command for: ora.asm 1 1
    2013-05-17 01:13:21.373: [    AGFW][3479152384] {0:0:118} ora.asm 1 1 state changed from: OFFLINE to: CLEANING
    2013-05-17 01:13:21.374: [ora.asm][3477051136] {0:0:118} [clean] (:CLSN00106:) clsn_agent::clean {                                                  
    2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] clean {                                                                            
    2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop_option stop mode immediate option 1
    2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop {                                                                  
    2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop original reason system do shutdown abort
    2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::resetConnection s_statusOfConnectionMap 00a87d90
    2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::resetConnection sid +ASM1 status  2                                
    2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] Gimh::check OH /u01/app/11.2.0/grid SID +ASM1                                      
    2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
    2013-05-17 01:13:21.375: [ora.asm][3477051136] {0:0:118} [clean] (:CLSN00006:)AsmAgent::check failed gimh state 0
    2013-05-17 01:13:21.376: [ora.asm][3477051136] {0:0:118} [clean] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
    2013-05-17 01:13:21.542: [ COMMCRS][3442890496]clsc_connect: (0x7f54b00993b0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

    2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] checkCrsStat 2 CLSCRS_STAT ret: 184
    2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] clsnUtils::error Exception type=2 string=

    2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] AsmAgent::checkCbk: Exception UserErrorException
    2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean]
    2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] (:CLSN00006:)InstAgent::check: return unplanned offline
    2013-05-17 01:13:21.543: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_query_ctx rc=0
    2013-05-17 01:13:21.543: [ USRTHRD][3477051136] {0:0:118} Gimh::destructor gimh_dest_inst_ctx rc=0
    2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::stopConnection
    2013-05-17 01:13:21.543: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::removeConnection connection count 0
    2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::removeConnection freed 0
    2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::stopConnection sid +ASM1 status  1    
    2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::check checkCounter 0 prev clsagfw_res_status 1 current clsagfw_res_status 1
    2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop shutdown mode: 4
    2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::getConnection 160 ERROR sid +ASM1 pool status 1                    
    2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] ConnectionPool::getConnection 260 pConnxn 00000000
    2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] InstAgent::stop pool pConnxn 00000000
    2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
    2013-05-17 01:13:21.544: [ora.asm][3477051136] {0:0:118} [clean] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/11.2.0/grid Crshome /u01/app/11.2.0/grid
    2013-05-17 01:13:21.544: [ora                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 10. Re: unable to start asm on one of the nodes
    Red8 Newbie
    Currently Being Moderated
    here is the out put that i have

    [root@KIFIYA-UBS-ORA-RAC1 bin]# ./crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4529: Cluster Synchronization Services is online
    CRS-4534: Cannot communicate with Event Manager
    [root@KIFIYA-UBS-ORA-RAC1 bin]# ./crsctl stat res -t -init
    --------------------------------------------------------------------------------
    NAME TARGET STATE SERVER STATE_DETAILS
    --------------------------------------------------------------------------------
    Cluster Resources
    --------------------------------------------------------------------------------
    ora.asm
    1 ONLINE OFFLINE
    ora.cluster_interconnect.haip
    1 ONLINE ONLINE kifiya-ubs-ora-rac1
    ora.crf
    1 ONLINE ONLINE kifiya-ubs-ora-rac1
    ora.crsd
    1 ONLINE OFFLINE
    ora.cssd
    1 ONLINE ONLINE kifiya-ubs-ora-rac1
    ora.cssdmonitor
    1 ONLINE ONLINE kifiya-ubs-ora-rac1
    ora.ctssd
    1 ONLINE ONLINE kifiya-ubs-ora-rac1 ACTIVE:0
    ora.diskmon
    1 OFFLINE OFFLINE
    ora.evmd
    1 ONLINE INTERMEDIATE kifiya-ubs-ora-rac1
    ora.gipcd
    1 ONLINE ONLINE kifiya-ubs-ora-rac1
    ora.gpnpd
    1 ONLINE ONLINE kifiya-ubs-ora-rac1
    ora.mdnsd
    1 ONLINE ONLINE kifiya-ubs-ora-rac1
    [root@KIFIYA-UBS-ORA-RAC1 bin]#
  • 11. Re: unable to start asm on one of the nodes
    rarain Explorer
    Currently Being Moderated
    As I seen in your log all devices have root privilege which is probably preventing Grid user to accessing physical disk and not able to start ASM instance which in turn make OCR devices inaccessible. Suggesting you to please assign appropriate privilege on the physical disks and retry to start CRS.

    [root@KIFIYA-UBS-ORA-RAC1 bin]# ls -l /dev/mapper/
    total 0
    crw-rw---- 1 root root 10, 58 May 17 10:16 control
    lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_DATA1 -> ../dm-2
    lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_DATA2 -> ../dm-8
    lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_FRA1 -> ../dm-7
    lrwxrwxrwx 1 root root 7 May 17 10:16 ORA_FRA2 -> ../dm-6
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK1 -> ../dm-0
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK2 -> ../dm-3
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK3 -> ../dm-5
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK4 -> ../dm-4
    lrwxrwxrwx 1 root root 7 May 17 10:16 VOTE_DSK5 -> ../dm-1

    Thanks

    Edited by: rarain on May 17, 2013 11:20 AM
  • 12. Re: unable to start asm on one of the nodes
    santysharma Explorer
    Currently Being Moderated
    Hi,

    Please check your grid user has right group privileges, you can compare with other nodes in cluster.

    Also try to manually login to ASM as grid user, most likely toy will get same error.



    Regards,
    Sharma
  • 13. Re: unable to start asm on one of the nodes
    Red8 Newbie
    Currently Being Moderated
    I have fixed it and here is how i done it. Actually it is very easy but it took 2 days for me to figure it out and the problem was my fello DB admin has changed
    the sqlnet.ora file authnetication to none. Because of this the ASM instance had a trouble bring it up.

    So what i did was, i commented out the autentication from the two sqlnet.ora files and it restarted the server, And everything worked perfectly

    Thanks for all your fox u tried to help, i greatly apperciated it.

    Thanks
    Rediet

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points