8 Replies Latest reply: May 3, 2012 3:58 AM by shayad RSS

    11gr2 clusterware not able to startup

    729036
      Hi Folks,
      i am having the issue on 2 node RAC , alll installation went fine and after reboot of 2 nodes clusterware not able to come online

      Oracle Clusterware active version on the cluster is [11.2.0.1.0]

      grid:ORA-B-R-D-RAC3$ uname -a
      Linux ORA-B-R-D-RAC3 2.6.18-238.5.1.el5 #1 SMP Mon Feb 21 05:52:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
      grid:ORA-B-R-D-RAC3$

      looks like some permissions issue after reboot. ASM not coming up so OCR/voting disk not accessible on ASM resulting in clusterware down.

      i tried to start ASM manually too, but looks like some permissions as ASM instance is pointing to oracle_base /u01/app/oracle rather then /u01/app/grid

      grid:ORA-B-R-D-RAC3$ sqlplus / as sysasm

      SQL*Plus: Release 11.2.0.1.0 Production on Sat Apr 28 09:18:27 2012

      Copyright (c) 1982, 2009, Oracle. All rights reserved.

      Connected to an idle instance.

      SQL> startup
      ORA-48141: error creating directory during ADR initialization [u01/app/oracle/diag/asm/+asm]
      ORA-48189: OS command to create directory failed
      Linux-x86_64 Error: 13: Permission denied
      Additional information: 2
      SQL>

      Tried to create manually the pfile for ASM instance and tried starting the ASM instance with pfile but got this error below

      grid:ORA-B-R-D-RAC3$ cat init+ASM1.ora
      asm_diskgroups='OCR_VOTING'
      #asm_diskstring='ORCL:*'
      asm_power_limit=1
      instance_type='asm'
      large_pool_size=12M
      remote_login_passwordfile='EXCLUSIVE'

      grid:ORA-B-R-D-RAC3$

      LMHB started with pid=12, OS id=8606
      Thu Apr 26 20:51:56 2012
      MMAN started with pid=13, OS id=8608
      Thu Apr 26 20:51:56 2012
      DBW0 started with pid=14, OS id=8610
      Thu Apr 26 20:51:56 2012
      LGWR started with pid=15, OS id=8612
      Thu Apr 26 20:51:56 2012
      CKPT started with pid=16, OS id=8614
      Thu Apr 26 20:51:56 2012
      SMON started with pid=17, OS id=8616
      Thu Apr 26 20:51:56 2012
      RBAL started with pid=18, OS id=8618
      Thu Apr 26 20:51:57 2012
      GMON started with pid=19, OS id=8620
      Thu Apr 26 20:51:57 2012
      MMON started with pid=20, OS id=8622
      Thu Apr 26 20:51:57 2012
      MMNL started with pid=21, OS id=8624
      lmon registered with NM - instance number 1 (internal mem no 0)
      Reconfiguration started (old inc 0, new inc 2)
      ASM instance
      List of instances:
      1 (myinst: 1)
      Global Resource Directory frozen
      * allocate domain 0, invalid = TRUE
      Communication channels reestablished
      Master broadcasted resource hash value bitmaps
      Non-local Process blocks cleaned out
      LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
      Set master node info
      Submitted all remote-enqueue requests
      Dwn-cvts replayed, VALBLKs dubious
      All grantable enqueues granted
      Post SMON to start 1st pass IR
      Submitted all GCS remote-cache requests
      Post SMON to start 1st pass IR
      Fix write in gcs resources
      Reconfiguration complete
      Thu Apr 26 20:51:57 2012
      LCK0 started with pid=22, OS id=8629
      Thu Apr 26 20:51:58 2012
      Shutting down instance (abort)
      License high water mark = 2
      USER (ospid: 8634): terminating the instance
      Instance terminated by USER, pid = 8634
      Thu Apr 26 20:51:58 2012
      Instance shutdown complete


      *grid:ORA-B-R-D-RAC3$ cat crsconfig_params
      *# $Header: has/install/crsconfig/crsconfig_params.sbs /main/17 2009/03/11 20:18:23 dpham Exp $
      #
      # crsconfig.lib
      #
      # Copyright (c) 2000, 2009, Oracle and/or its affiliates.All rights reserved.
      #
      # NAME
      # crsconfig_params.sbs - Installer variables required for root config
      #

      #
      # crsconfig_params.sbs -
      #
      # ==========================================================

      SILENT=false
      ORACLE_OWNER=grid
      ORA_DBA_GROUP=oinstall
      ORA_ASM_GROUP=oinstall
      LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8

      ORACLE_HOME=/u01/app/11.2.0/grid
      ORACLE_BASE=/u01/app/grid

      JREDIR=/u01/app/11.2.0/grid/jdk/jre/
      JLIBDIR=/u01/app/11.2.0/grid/jlib
      NETCFGJAR_NAME=netcfg.jar
      EWTJAR_NAME=ewt3.jar
      JEWTJAR_NAME=jewt4.jar
      SHAREJAR_NAME=share.jar
      HELPJAR_NAME=help4.jar
      EMBASEJAR_NAME=oemlt.jar

      VNDR_CLUSTER=false
      OCR_LOCATIONS=NO_VAL
      CLUSTER_NAME=ORA-B-R-cluster
      HOST_NAME_LIST=ORA-B-R-D-RAC3,ORA-B-R-D-RAC4
      NODE_NAME_LIST=ORA-B-R-D-RAC3,ORA-B-R-D-RAC4
      PRIVATE_NAME_LIST=
      VOTING_DISKS=NO_VAL
      #VF_DISCOVERY_STRING=%s_vfdiscoverystring%
      ASM_UPGRADE=false
      ASM_SPFILE=
      ASM_DISK_GROUP=OCR_VOTING
      ASM_DISCOVERY_STRING=
      ASM_DISKS=ORCL:CRSDISK1,ORCL:CRSDISK2
      ASM_REDUNDANCY=EXTERNAL
      CRS_STORAGE_OPTION=1
      CSS_LEASEDURATION=400
      CRS_NODEVIPS='ORA-B-R-D-RAC3-VIP/255.255.254.0/eth0,ORA-B-R-D-RAC4-VIP/255.255.254.0/eth0'
      NODELIST=ORA-B-R-D-RAC3,ORA-B-R-D-RAC4
      NETWORKS="eth0"/10.193.58.0:public,"eth1"/192.168.1.0:cluster_interconnect
      SCAN_NAME=ORA-B-R-D-RAC2-SCAN.sysdev.adroot.bmogc.net
      SCAN_PORT=1521
      GPNP_PA=
      OCFS_CONFIG=

      # GNS consts
      GNS_CONF=false
      GNS_ADDR_LIST=
      GNS_DOMAIN_LIST=
      GNS_ALLOW_NET_LIST=
      GNS_DENY_NET_LIST=
      GNS_DENY_ITF_LIST=

      #### Required by OUI add node
      NEW_HOST_NAME_LIST=
      NEW_NODE_NAME_LIST=
      NEW_PRIVATE_NAME_LIST=
      NEW_NODEVIPS='ORA-B-R-D-RAC3-VIP/255.255.254.0/eth0,ORA-B-R-D-RAC4-VIP/255.255.254.0/eth0'

      ############### OCR constants
      # GPNPCONFIGDIR is handled differently in dev (T_HAS_WORK for all)
      # GPNPGCONFIGDIR in dev expands to T_HAS_WORK_GLOBAL
      GPNPCONFIGDIR=$ORACLE_HOME
      GPNPGCONFIGDIR=$ORACLE_HOME
      OCRLOC=
      OLRLOC=
      OCRID=
      CLUSTER_GUID=

      CLSCFG_MISSCOUNT=


      checked the ASM headers and ASM disks with KFED all are valid disks

      crsd.log is as below



      2012-04-26 20:04:08.505: [    GPnP][856851168]clsgpnp_InitCKProviders: [at clsgpnp0.c:3891] Init gpnp local security key provider 1 of 2: file wallet (LSKP-FSW) OK
      2012-04-26 20:04:08.505: [    GPnP][856851168]clsgpnp_InitCKProviders: [at clsgpnp0.c:3897] Init gpnp local security key proveders 2 of 2: OLR wallet (LSKP-CLSW-OLR)
      [   CLWAL][856851168]clsw_Initialize: OLR initlevel [30000]
      2012-04-26 20:04:08.506: [ CRSMAIN][1093040448] Policy Engine is not initialized yet!
      2012-04-26 20:04:08.515: [    GPnP][856851168]clsgpnp_InitCKProviders: [at clsgpnp0.c:3919] Init gpnp local security key provider 2 of 2: OLR wallet (LSKP-CLSW-OLR) OK
      2012-04-26 20:04:08.516: [    GPnP][856851168]clsgpnp_getCK: [at clsgpnp0.c:1950] <Get gpnp security keys (wallet) for id:1,typ;7. (2 providers - fatal if all fail)
      2012-04-26 20:04:08.516: [    GPnP][856851168]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:498] req_id=1 ck_prov_id=1 wallet path: /u01/app/11.2.0/grid/gpnp/ora-b-r-d-rac3/wallets/peer/
      2012-04-26 20:04:08.525: [    GPnP][856851168]clsgpnpwu_walletfopen: [at clsgpnpwu.c:494] Opened SSO wallet: '/u01/app/11.2.0/grid/gpnp/ora-b-r-d-rac3/wallets/peer/cwallet.sso'
      2012-04-26 20:04:08.525: [    GPnP][856851168]clsgpnp_getCK: [at clsgpnp0.c:1965] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1 of 2 (LSKP-FSW(1))
      2012-04-26 20:04:08.525: [    GPnP][856851168]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
      2012-04-26 20:04:08.528: [    GPnP][856851168]clsgpnp_getCK: [at clsgpnp0.c:1950] <Get gpnp security keys (wallet) for id:1,typ;4. (2 providers - fatal if all fail)
      2012-04-26 20:04:08.528: [    GPnP][856851168]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:498] req_id=1 ck_prov_id=1 wallet path: /u01/app/11.2.0/grid/gpnp/ora-b-r-d-rac3/wallets/peer/
      2012-04-26 20:04:08.535: [    GPnP][856851168]clsgpnpwu_walletfopen: [at clsgpnpwu.c:494] Opened SSO wallet: '/u01/app/11.2.0/grid/gpnp/ora-b-r-d-rac3/wallets/peer/cwallet.sso'
      2012-04-26 20:04:08.536: [    GPnP][856851168]clsgpnp_getCK: [at clsgpnp0.c:1965] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1 of 2 (LSKP-FSW(1))
      2012-04-26 20:04:08.536: [    GPnP][856851168]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
      2012-04-26 20:04:08.536: [    GPnP][856851168]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=5077, tl=3, f=0
      2012-04-26 20:04:08.552: [GIPCXCPT][856851168] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)
      2012-04-26 20:04:08.554: [GIPCXCPT][856851168] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)
      2012-04-26 20:04:08.563: [  OCRASM][856851168]proprasmo: Error in open/create file in dg [OCR_VOTING]
      [  OCRASM][856851168]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
      ORA-15077: could not locate ASM instance serving a required diskgroup

      2012-04-26 20:04:08.565: [  OCRASM][856851168]proprasmo: kgfoCheckMount returned [7]
      2012-04-26 20:04:08.565: [  OCRASM][856851168]proprasmo: The ASM instance is down
      2012-04-26 20:04:08.565: [  OCRRAW][856851168]proprioo: Failed to open [+OCR_VOTING]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
      2012-04-26 20:04:08.565: [  OCRRAW][856851168]proprioo: No OCR/OLR devices are usable
      2012-04-26 20:04:08.565: [  OCRASM][856851168]proprasmcl: asmhandle is NULL
      2012-04-26 20:04:08.565: [  OCRRAW][856851168]proprinit: Could not open raw device
      2012-04-26 20:04:08.565: [  OCRASM][856851168]proprasmcl: asmhandle is NULL
      2012-04-26 20:04:08.565: [  OCRAPI][856851168]a_init:16!: Backend init unsuccessful : [26]
      2012-04-26 20:04:08.565: [  CRSOCR][856851168] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
      ORA-15077: could not locate ASM instance serving a required diskgroup
      ] [7]
      2012-04-26 20:04:08.565: [    CRSD][856851168][PANIC] CRSD exiting: Could not init OCR, code: 26
      2012-04-26 20:04:08.566: [    CRSD][856851168] Done.
        • 1. Re: 11gr2 clusterware not able to startup
          hitgon
          Hi,

          LogFile........................
          2012-04-26 20:04:08.565: [ CRSOCR][856851168] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

          Please check your shared storage partition on each nodes with same configuration path/name/size and
          assign the permission on each nodes

          Example
          You have to given the proper permission to shared storage partition on each nodes inside the /etc/rc.local file

          Add the following commands to the /etc/rc.local file.

          /etc/rc.local
          chmod 660 /dev/sdh*
          chmod 660 /dev/sdi*
          chmod 777 /dev/sdb*
          chmod 777 /dev/sdd*
          chmod 777 /dev/sde*
          chmod 777 /dev/sdf*
          chmod 777 /dev/sdg*

          chown -R oracle:oinstall /dev/sdh*
          chown -R oracle:oinstall /dev/sdi*
          chown -R oracle:oinstall /dev/sdb*
          chown -R oracle:oinstall /dev/sdd*
          chown -R oracle:oinstall /dev/sde*
          chown -R oracle:oinstall /dev/sdf*
          chown -R oracle:oinstall /dev/sdg*

          Please set the proper permission and try to start the asm instance
          Regards
          Hitgon

          Edited by: hitgon on Apr 28, 2012 7:00 PM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
          • 2. Re: 11gr2 clusterware not able to startup
            729036
            i have another 2 node RAC primary which has exactly same permissons and working fine

            on PRIMARY (no issues)

            oracle:ORA-B-R-D-RAC1$ ls -alrt sd*
            brw-r----- 1 root disk 8, 32 Apr 25 14:29 sdc
            brw-r----- 1 root disk 8, 17 Apr 25 14:29 sdb1
            brw-r----- 1 root disk 8, 16 Apr 25 14:29 sdb
            brw-r----- 1 root disk 8, 2 Apr 25 14:29 sda2
            brw-r----- 1 root disk 8, 0 Apr 25 14:29 sda
            brw-r----- 1 root disk 8, 1 Apr 25 14:29 sda1
            brw-r----- 1 root disk 8, 33 Apr 25 14:29 sdc1
            brw-r----- 1 root disk 8, 34 Apr 25 14:29 sdc2
            brw-r----- 1 root disk 8, 36 Apr 25 14:29 sdc4
            brw-r----- 1 root disk 8, 37 Apr 25 14:29 sdc5
            brw-r----- 1 root disk 8, 35 Apr 25 14:29 sdc3
            brw-r----- 1 root disk 8, 38 Apr 25 14:29 sdc6
            brw-r----- 1 root disk 8, 41 Apr 25 14:29 sdc9
            brw-r----- 1 root disk 8, 40 Apr 25 14:29 sdc8
            brw-r----- 1 root disk 8, 39 Apr 25 14:29 sdc7
            brw-r----- 1 root disk 8, 42 Apr 25 14:29 sdc10
            brw-r----- 1 root disk 8, 43 Apr 25 14:29 sdc11
            brw-r----- 1 root disk 8, 44 Apr 25 14:29 sdc12
            brw-r----- 1 root disk 8, 45 Apr 25 14:29 sdc13
            brw-r----- 1 root disk 8, 46 Apr 25 14:29 sdc14
            brw-r----- 1 root disk 8, 47 Apr 25 14:29 sdc15

            racle:ORA-B-R-D-RAC1$ /sbin/blkid |grep oracleasm
            /dev/sdc15: LABEL="FRAVOL6" TYPE="oracleasm"
            /dev/sdc14: LABEL="FRAVOL5" TYPE="oracleasm"
            /dev/sdc13: LABEL="FRAVOL4" TYPE="oracleasm"
            /dev/sdc12: LABEL="FRAVOL3" TYPE="oracleasm"
            /dev/sdc11: LABEL="FRAVOL2" TYPE="oracleasm"
            /dev/sdc10: LABEL="FRAVOL1" TYPE="oracleasm"
            /dev/sdc9: LABEL="DATAVOL5" TYPE="oracleasm"
            /dev/sdc8: LABEL="DATAVOL4" TYPE="oracleasm"
            /dev/sdc7: LABEL="DATAVOL3" TYPE="oracleasm"
            /dev/sdc6: LABEL="DATAVOL2" TYPE="oracleasm"
            /dev/sdc5: LABEL="DATAVOL1" TYPE="oracleasm"
            /dev/sdc3: LABEL="CRSV3" TYPE="oracleasm"
            /dev/sdc2: LABEL="CRSV2" TYPE="oracleasm"
            /dev/sdc1: LABEL="CRSV1" TYPE="oracleasm"
            oracle:ORA-B-R-D-RAC1$


            FOR PROBLEM rac both nodes

            oracle:ORA-B-R-D-RAC3$ ls -alrt /dev/sd*
            brw-r----- 1 root disk 8, 32 Apr 26 20:00 /dev/sdc
            brw-r----- 1 root disk 8, 17 Apr 26 20:00 /dev/sdb1
            brw-r----- 1 root disk 8, 16 Apr 26 20:00 /dev/sdb
            brw-r----- 1 root disk 8, 2 Apr 26 20:00 /dev/sda2
            brw-r----- 1 root disk 8, 0 Apr 26 20:00 /dev/sda
            brw-r----- 1 root disk 8, 1 Apr 26 20:00 /dev/sda1
            brw-r----- 1 root disk 8, 33 Apr 26 20:01 /dev/sdc1
            brw-r----- 1 root disk 8, 34 Apr 26 20:01 /dev/sdc2
            brw-r----- 1 root disk 8, 36 Apr 26 20:01 /dev/sdc4
            brw-r----- 1 root disk 8, 39 Apr 26 20:01 /dev/sdc7
            brw-r----- 1 root disk 8, 41 Apr 26 20:01 /dev/sdc9
            brw-r----- 1 root disk 8, 42 Apr 26 20:01 /dev/sdc10
            brw-r----- 1 root disk 8, 44 Apr 26 20:01 /dev/sdc12
            brw-r----- 1 root disk 8, 38 Apr 26 20:01 /dev/sdc6
            brw-r----- 1 root disk 8, 40 Apr 26 20:01 /dev/sdc8
            brw-r----- 1 root disk 8, 43 Apr 26 20:01 /dev/sdc11
            brw-r----- 1 root disk 8, 35 Apr 26 20:01 /dev/sdc3
            brw-r----- 1 root disk 8, 37 Apr 26 20:01 /dev/sdc5

            oracle:ORA-B-R-D-RAC3$ /sbin/blkid |grep oracleasm
            /dev/sdc2: LABEL="CRSDISK2" TYPE="oracleasm"
            /dev/sdc1: LABEL="CRSDISK1" TYPE="oracleasm"
            /dev/sdc3: LABEL="DATAVOL1" TYPE="oracleasm"
            /dev/sdc5: LABEL="DATAVOL2" TYPE="oracleasm"
            /dev/sdc6: LABEL="DATAVOL3" TYPE="oracleasm"
            /dev/sdc7: LABEL="DATAVOL4" TYPE="oracleasm"
            /dev/sdc8: LABEL="DATAVOL5" TYPE="oracleasm"
            /dev/sdc9: LABEL="FRAVOL1" TYPE="oracleasm"
            /dev/sdc10: LABEL="FRAVOL2" TYPE="oracleasm"
            /dev/sdc11: LABEL="FRAVOL3" TYPE="oracleasm"
            /dev/sdc12: LABEL="FRAVOL4" TYPE="oracleasm"
            oracle:ORA-B-R-D-RAC3$
            • 3. Re: 11gr2 clusterware not able to startup
              user503699
              ricky wrote:
              Hi Folks,
              i am having the issue on 2 node RAC , alll installation went fine and after reboot of 2 nodes clusterware not able to come online
              This may not be relevant to your problem (and may not be an issue at all so feel free to ignore) but the following settings in your param file look unusual.
              ORACLE_HOME=/u01/app/11.2.0/grid
              ORACLE_BASE=/u01/app/grid
              Generally, ORACLE_HOME path is contained within directory defined as ORACLE_BASE. Is that just a "best practice" and not a mandatory requirement?

              Turns out above is indeed a non-issue and in fact a requirement. Sorry about it.
              BTW, are the definitions of GRID_BASE, GRID_HOME, ORACLE_BASE and ORACLE_HOME environment variables correct in the .profile (or .bash_profile) of your GRID user?

              Edited by: user503699 on Apr 29, 2012 2:38 AM
              • 4. Re: 11gr2 clusterware not able to startup
                788605
                Clusterware is not starting because of your issue in asm instance, isolate the problem one by one. Check the cause of asm instance not starting up. Can you post your asm log when doing manual startup of asm?
                • 5. Re: 11gr2 clusterware not able to startup
                  729036
                  Thanks for your help, here is the log for ASM when tries to start ASM instance manually with pfile

                  grid:ORA-B-R-D-RAC3$ cat init+asm.ora
                  +ASM1.__oracle_base='/u01/app/grid'#ORACLE_BASE set from in memory value
                  +ASM2.__oracle_base='/u01/app/grid'#ORACLE_BASE set from in memory value
                  +ASM2.asm_diskgroups='DATA','FRA'#Manual Mount
                  +ASM1.asm_diskgroups='OCC_VOTING'
                  *.asm_power_limit=1
                  *.diagnostic_dest='/u01/app/grid'
                  *.instance_type='asm'
                  *.large_pool_size=12M
                  +ASM1.processes=200
                  *.remote_login_passwordfile='EXCLUSIVE


                  grid:$ sqlplus / as sysasm

                  SQL*Plus: Release 11.2.0.1.0 Production on Sun Apr 29 20:37:53 2012

                  Copyright (c) 1982, 2009, Oracle. All rights reserved.

                  Connected to an idle instance.

                  SQL> startup pfile='/u01/app/grid/init+asm.ora';
                  ORA-03113: end-of-file on communication channel
                  SQL> exit
                  Disconnected



                  Starting up:
                  Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
                  With the Real Application Clusters and Automatic Storage Management options.
                  Using parameter settings in client-side pfile /u01/app/grid/init+asm.ora on machine ORA-B-R-D-RAC3
                  System parameters with non-default values:
                  processes = 200
                  large_pool_size = 12M
                  instance_type = "asm"
                  remote_login_passwordfile= "EXCLUSIVE"
                  asm_diskgroups = "OCC_VOTING"
                  asm_power_limit = 1
                  diagnostic_dest = "/u01/app/grid"
                  Cluster communication is configured to use the following interface(s) for this instance
                  192.168.1.111
                  cluster interconnect IPC version:Oracle UDP/IP (generic)
                  IPC Vendor 1 proto 2
                  Sun Apr 29 20:38:13 2012
                  PMON started with pid=2, OS id=27851
                  Sun Apr 29 20:38:13 2012
                  VKTM started with pid=3, OS id=27853 at elevated priority
                  VKTM running at (10)millisec precision with DBRM quantum (100)ms
                  Sun Apr 29 20:38:13 2012
                  GEN0 started with pid=4, OS id=27857
                  Sun Apr 29 20:38:13 2012
                  DIAG started with pid=5, OS id=27859
                  Sun Apr 29 20:38:13 2012
                  PING started with pid=6, OS id=27861
                  Sun Apr 29 20:38:13 2012
                  PSP0 started with pid=7, OS id=27863
                  Sun Apr 29 20:38:13 2012
                  DIA0 started with pid=8, OS id=27865
                  Sun Apr 29 20:38:13 2012
                  LMON started with pid=9, OS id=27867
                  Sun Apr 29 20:38:16 2012
                  LMD0 started with pid=10, OS id=27869
                  Sun Apr 29 20:38:16 2012
                  LMS0 started with pid=11, OS id=27873 at elevated priority
                  Sun Apr 29 20:38:16 2012
                  LMHB started with pid=12, OS id=27877
                  Sun Apr 29 20:38:16 2012
                  MMAN started with pid=13, OS id=27880
                  Sun Apr 29 20:38:16 2012
                  DBW0 started with pid=14, OS id=27882
                  Sun Apr 29 20:38:16 2012
                  LGWR started with pid=15, OS id=27884
                  Sun Apr 29 20:38:16 2012
                  CKPT started with pid=16, OS id=27886
                  Sun Apr 29 20:38:16 2012
                  SMON started with pid=17, OS id=27888
                  Sun Apr 29 20:38:16 2012
                  RBAL started with pid=18, OS id=27890
                  Sun Apr 29 20:38:16 2012
                  GMON started with pid=19, OS id=27892
                  Sun Apr 29 20:38:17 2012
                  MMON started with pid=20, OS id=27894
                  Sun Apr 29 20:38:17 2012
                  MMNL started with pid=21, OS id=27896
                  lmon registered with NM - instance number 1 (internal mem no 0)
                  Reconfiguration started (old inc 0, new inc 2)
                  ASM instance
                  List of instances:
                  1 (myinst: 1)
                  Global Resource Directory frozen
                  * allocate domain 0, invalid = TRUE
                  Communication channels reestablished
                  Master broadcasted resource hash value bitmaps
                  Non-local Process blocks cleaned out
                  LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
                  Set master node info
                  Submitted all remote-enqueue requests
                  Dwn-cvts replayed, VALBLKs dubious
                  All grantable enqueues granted
                  Post SMON to start 1st pass IR
                  Submitted all GCS remote-cache requests
                  Post SMON to start 1st pass IR
                  Fix write in gcs resources
                  Reconfiguration complete
                  Sun Apr 29 20:38:17 2012
                  LCK0 started with pid=22, OS id=27901
                  Sun Apr 29 20:38:18 2012
                  Shutting down instance (abort)
                  License high water mark = 2
                  USER (ospid: 27906): terminating the instance
                  Instance terminated by USER, pid = 27906
                  Sun Apr 29 20:38:19 2012
                  Instance shutdown complete
                  • 6. Re: 11gr2 clusterware not able to startup
                    729036
                    Thanks ... for your help. Here is the bash profile


                    .bash_profile

                    export PS1="`whoami`:`hostname`$ " >> .bash_profile
                    export EDITOR=vi
                    export JAVA_HOME=/usr/local/java
                    export ORACLE_BASE=/u01/app/grid
                    export ORACLE_HOME=/u01/app/11.2.0/grid
                    export ORACLE_SID=+ASM1
                    export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin:/u01/app/11.2.0/grid/bin:/home/grid/scripts
                    • 7. Re: 11gr2 clusterware not able to startup
                      onedbguru
                      what username starts up grid? is it username grid?

                      Given:
                      ORA-48141: error creating directory during ADR initialization [u01/app/oracle/diag/asm/+asm]
                      ORA-48189: OS command to create directory failed
                      Linux-x86_64 Error: 13: Permission denied
                      Additional information: 2

                      means that user grid does not have create directory privs on the ORACLE_BASE /u01/app/oracle

                      chmod 777 /u01/app/oracle/diag

                      or - fix your grid environment to point to a grid accessible directory structure.

                      Looking at your bash profile you might want to read the following a bit more carefully. particularly: "C.1.2.2 Understanding Oracle Base and Grid Infrastructure Directories"

                      http://docs.oracle.com/cd/E11882_01/install.112/e24615/concepts.htm#BABIJFHC

                      also, if you don't want to edit the bash profile on every node, the "case statement" is your friend.

                      For ASM to work, the ownership of the devices must be <grid_user>:<grid_group> where the oracle user is also in the grid_group and the files have 660 permissions.
                      • 8. Re: 11gr2 clusterware not able to startup
                        shayad
                        SQL> startup
                        ORA-48141: error creating directory during ADR initialization [u01/app/oracle/diag/asm/+asm]
                        ORA-48189: OS command to create directory failed
                        Linux-x86_64 Error: 13: Permission denied
                        Additional information: 2

                        For the same set the permissions for the directories..
                        1. From root User run
                        */etc/init.d/oracle asm configure*
                        User name : grid/oracle
                        Group : oinstall
                        : Y
                        : Y

                        2. Set the Permissions
                        chown -R grid:oinstall /u01/app/oracle (if You are using grid user, if not using grid then use oracle instead of grid.)
                        chmod -R 775 /u01/app/oracle

                        Edited by: shayad on May 3, 2012 1:55 AM

                        Edited by: shayad on May 3, 2012 1:57 AM

                        Edited by: shayad on May 3, 2012 1:58 AM