4 Replies Latest reply: Jul 14, 2013 10:20 PM by 993159 RSS

    root.ah fails on 2nd node(rac2) with [ ORA-15018,ORA-15017,ORA-15003 ]

    993159

      Hi All,

       

      I m trying to setup 11gR2 Grid installation on two-node Rac . When it comes to running root.sh on second node (i.e. rac2) it fails with below error. Could please anyone help me out. This is my 3rd attempt and all fails with below errors on node 2.

       

      rac2:

      [root@rac2 grid_home]# ./root.sh

      Running Oracle 11g root.sh script...

       

       

      The following environment variables are set as:

          ORACLE_OWNER= grid

          ORACLE_HOME=  /u01/grid_home

       

       

      Enter the full pathname of the local bin directory: [/usr/local/bin]:

         Copying dbhome to /usr/local/bin ...

         Copying oraenv to /usr/local/bin ...

         Copying coraenv to /usr/local/bin ...

       

       

       

       

      Creating /etc/oratab file...

      Entries will be added to the /etc/oratab file as needed by

      Database Configuration Assistant when a database is created

      Finished running generic part of root.sh script.

      Now product-specific root actions will be performed.

      2013-07-10 18:53:15: Parsing the host name

      2013-07-10 18:53:15: Checking for super user privileges

      2013-07-10 18:53:15: User has super user privileges

      Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params

      Creating trace directory

      LOCAL ADD MODE

      Creating OCR keys for user 'root', privgrp 'root'..

      Operation successful.

      Adding daemon to inittab

      CRS-4123: Oracle High Availability Services has been started.

      ohasd is starting

      CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'

      CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'

      CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded

      CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded

      CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'

      CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded

      CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'

      CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded

      CRS-2672: Attempting to start 'ora.cssd' on 'rac2'

      CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'

      CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded

      CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded

      CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'

      CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded

       

       

      DiskGroup CRS creation failed with the following message:

      ORA-15018: diskgroup cannot be created

      ORA-15017: diskgroup "CRS" cannot be mounted

      ORA-15003: diskgroup "CRS" already mounted in another lock name space

       

       

       

       

      Configuration of ASM failed, see logs for details

      Did not succssfully configure and start ASM

      CRS-2500: Cannot stop resource 'ora.crsd' as it is not running

      CRS-4000: Command Stop failed, or completed with errors.

      Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init

      Stop of resource "ora.crsd -init" failed

      Failed to stop CRSD

      CRS-2673: Attempting to stop 'ora.asm' on 'rac2'

      CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded

      CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'

      CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded

      CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac2'

      CRS-2677: Stop of 'ora.cssdmonitor' on 'rac2' succeeded

      CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'

      CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded

      CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'

      CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded

      CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'

      CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded

      CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'

      CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded

      Initial cluster configuration failed.  See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details

      [root@rac2 grid_home]#

       

       

       

      rac2  alertrac2.log

       

      [root@rac2 rac2]# cat -n alertrac2.log

           1  Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.

           2  2013-07-10 18:53:16.145

           3  [client(13088)]CRS-2106:The OLR location /u01/grid_home/cdata/rac2.olr is inaccessible. Details in /u01/grid_home/log/rac2/client/ocrconfig_13088.log.

           4  2013-07-10 18:53:16.228

           5  [client(13088)]CRS-2101:The OLR was formatted using version 3.

           6  2013-07-10 18:53:31.734

           7  [ohasd(13132)]CRS-2112:The OLR service started on node rac2.

           8  2013-07-10 18:53:31.893

           9  [ohasd(13132)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.

          10  2013-07-10 18:53:53.762

          11  [ohasd(13132)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

          12  2013-07-10 18:53:55.381

          13  [cssd(14409)]CRS-1713:CSSD daemon is started in exclusive mode

          14  2013-07-10 18:54:01.530

          15  [cssd(14409)]CRS-1709:Lease acquisition failed for node rac2 because no voting file has been configured; Details at (:CSSNM00031:) in /u01/grid_home/log/rac2/cssd/ocssd.log

          16  2013-07-10 18:54:19.113

          17  [cssd(14409)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .

          18  2013-07-10 18:54:19.910

          19  [ctssd(14465)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.

          20  2013-07-10 18:54:19.920

          21  [ctssd(14465)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.

          22  2013-07-10 18:54:20.903

          23  [ctssd(14465)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.

          24  [client(14715)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.

          25  [client(14719)]CRS-10001:ACFS-9322: done.

          26  2013-07-10 18:54:47.104

          27  [ctssd(14465)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user

          28  2013-07-10 18:54:55.837

          29  [cssd(14409)]CRS-1603:CSSD on node rac2 shutdown by user.

       

      rac2 rootcrs logfile

       

      [root@rac2 rac2]# cat  /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log

      2013-07-10 18:53:15: The configuration parameter file /u01/grid_home/crs/install/crsconfig_params is valid

      2013-07-10 18:53:15: Checking for super user privileges

      2013-07-10 18:53:15: User has super user privileges

      2013-07-10 18:53:15: ### Printing the configuration values from files:

      2013-07-10 18:53:15:    /u01/grid_home/crs/install/crsconfig_params

      2013-07-10 18:53:15:    /u01/grid_home/crs/install/s_crsconfig_defs

      2013-07-10 18:53:15: ASM_DISCOVERY_STRING=

      2013-07-10 18:53:15: ASM_DISKS=ORCL:CRS1

      2013-07-10 18:53:15: ASM_DISK_GROUP=CRS

      2013-07-10 18:53:15: ASM_REDUNDANCY=EXTERNAL

      2013-07-10 18:53:15: ASM_SPFILE=

      2013-07-10 18:53:15: ASM_UPGRADE=false

      2013-07-10 18:53:15: CLSCFG_MISSCOUNT=

      2013-07-10 18:53:15: CLUSTER_GUID=

      2013-07-10 18:53:15: CLUSTER_NAME=rac-scan

      2013-07-10 18:53:15: CRS_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'

      2013-07-10 18:53:15: CRS_STORAGE_OPTION=1

      2013-07-10 18:53:15: CSS_LEASEDURATION=400

      2013-07-10 18:53:15: DIRPREFIX=

      2013-07-10 18:53:15: DISABLE_OPROCD=0

      2013-07-10 18:53:15: EMBASEJAR_NAME=oemlt.jar

      2013-07-10 18:53:15: EWTJAR_NAME=ewt3.jar

      2013-07-10 18:53:15: EXTERNAL_ORACLE_BIN=/opt/oracle/bin

      2013-07-10 18:53:15: GNS_ADDR_LIST=

      2013-07-10 18:53:15: GNS_ALLOW_NET_LIST=

      2013-07-10 18:53:15: GNS_CONF=false

      2013-07-10 18:53:15: GNS_DENY_ITF_LIST=

      2013-07-10 18:53:15: GNS_DENY_NET_LIST=

      2013-07-10 18:53:15: GNS_DOMAIN_LIST=

      2013-07-10 18:53:15: GPNPCONFIGDIR=/u01/grid_home

      2013-07-10 18:53:15: GPNPGCONFIGDIR=/u01/grid_home

      2013-07-10 18:53:15: GPNP_PA=

      2013-07-10 18:53:15: HELPJAR_NAME=help4.jar

      2013-07-10 18:53:15: HOST_NAME_LIST=rac1,rac2

      2013-07-10 18:53:15: ID=/etc/init.d

      2013-07-10 18:53:15: INIT=/sbin/init

      2013-07-10 18:53:15: IT=/etc/inittab

      2013-07-10 18:53:15: JEWTJAR_NAME=jewt4.jar

      2013-07-10 18:53:15: JLIBDIR=/u01/grid_home/jlib

      2013-07-10 18:53:15: JREDIR=/u01/grid_home/jdk/jre/

      2013-07-10 18:53:15: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8

      2013-07-10 18:53:15: MSGFILE=/var/adm/messages

      2013-07-10 18:53:15: NETCFGJAR_NAME=netcfg.jar

      2013-07-10 18:53:15: NETWORKS="eth0"/192.168.0.0:public,"eth1"/192.168.1.0:cluster_interconnect

      2013-07-10 18:53:15: NEW_HOST_NAME_LIST=

      2013-07-10 18:53:15: NEW_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'

      2013-07-10 18:53:15: NEW_NODE_NAME_LIST=

      2013-07-10 18:53:15: NEW_PRIVATE_NAME_LIST=

      2013-07-10 18:53:15: NODELIST=rac1,rac2

      2013-07-10 18:53:15: NODE_NAME_LIST=rac1,rac2

      2013-07-10 18:53:15: OCFS_CONFIG=

      2013-07-10 18:53:15: OCRCONFIG=/etc/oracle/ocr.loc

      2013-07-10 18:53:15: OCRCONFIGDIR=/etc/oracle

      2013-07-10 18:53:15: OCRID=

      2013-07-10 18:53:15: OCRLOC=ocr.loc

      2013-07-10 18:53:15: OCR_LOCATIONS=NO_VAL

      2013-07-10 18:53:15: OLASTGASPDIR=/etc/oracle/lastgasp

      2013-07-10 18:53:15: OLRCONFIG=/etc/oracle/olr.loc

      2013-07-10 18:53:15: OLRCONFIGDIR=/etc/oracle

      2013-07-10 18:53:15: OLRLOC=olr.loc

      2013-07-10 18:53:15: OPROCDCHECKDIR=/etc/oracle/oprocd/check

      2013-07-10 18:53:15: OPROCDDIR=/etc/oracle/oprocd

      2013-07-10 18:53:15: OPROCDFATALDIR=/etc/oracle/oprocd/fatal

      2013-07-10 18:53:15: OPROCDSTOPDIR=/etc/oracle/oprocd/stop

      2013-07-10 18:53:15: ORACLE_BASE=/u01/11.2.0

      2013-07-10 18:53:15: ORACLE_HOME=/u01/grid_home

      2013-07-10 18:53:15: ORACLE_OWNER=grid

      2013-07-10 18:53:15: ORA_ASM_GROUP=asmadmin

      2013-07-10 18:53:15: ORA_DBA_GROUP=oinstall

      2013-07-10 18:53:15: PRIVATE_NAME_LIST=

      2013-07-10 18:53:15: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d

      2013-07-10 18:53:15: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d

      2013-07-10 18:53:15: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d

      2013-07-10 18:53:15: RC_KILL=K19

      2013-07-10 18:53:15: RC_KILL_OLD=K96

      2013-07-10 18:53:15: RC_START=S96

      2013-07-10 18:53:15: SCAN_NAME=rac-scan.naveed.com

      2013-07-10 18:53:15: SCAN_PORT=1521

      2013-07-10 18:53:15: SCRBASE=/etc/oracle/scls_scr

      2013-07-10 18:53:15: SHAREJAR_NAME=share.jar

      2013-07-10 18:53:15: SILENT=false

      2013-07-10 18:53:15: SO_EXT=so

      2013-07-10 18:53:15: SRVCFGLOC=srvConfig.loc

      2013-07-10 18:53:15: SRVCONFIG=/var/opt/oracle/srvConfig.loc

      2013-07-10 18:53:15: SRVCONFIGDIR=/var/opt/oracle

      2013-07-10 18:53:15: VNDR_CLUSTER=false

      2013-07-10 18:53:15: VOTING_DISKS=NO_VAL

      2013-07-10 18:53:15: ### Printing other configuration values ###

      2013-07-10 18:53:15: CLSCFG_EXTRA_PARMS=

      2013-07-10 18:53:15: CRSDelete=0

      2013-07-10 18:53:15: CRSPatch=0

      2013-07-10 18:53:15: DEBUG=

      2013-07-10 18:53:15: DOWNGRADE=

      2013-07-10 18:53:15: HAS_GROUP=oinstall

      2013-07-10 18:53:15: HAS_USER=root

      2013-07-10 18:53:15: HOST=rac2

      2013-07-10 18:53:15: IS_SIHA=0

      2013-07-10 18:53:15: OLR_DIRECTORY=/u01/grid_home/cdata

      2013-07-10 18:53:15: OLR_LOCATION=/u01/grid_home/cdata/rac2.olr

      2013-07-10 18:53:15: ORA_CRS_HOME=/u01/grid_home

      2013-07-10 18:53:15: SUPERUSER=root

      2013-07-10 18:53:15: UPGRADE=

      2013-07-10 18:53:15: VF_DISCOVERY_STRING=

      2013-07-10 18:53:15: addfile=/u01/grid_home/crs/install/crsconfig_addparams

      2013-07-10 18:53:15: crscfg_trace=1

      2013-07-10 18:53:15: crscfg_trace_file=/u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log

      2013-07-10 18:53:15: hosts=

      2013-07-10 18:53:15: oldcrshome=

      2013-07-10 18:53:15: oldcrsver=

      2013-07-10 18:53:15: osdfile=/u01/grid_home/crs/install/s_crsconfig_defs

      2013-07-10 18:53:15: parameters_valid=1

      2013-07-10 18:53:15: paramfile=/u01/grid_home/crs/install/crsconfig_params

      2013-07-10 18:53:15: platform_family=unix

      2013-07-10 18:53:15: srvctl_trc_suff=0

      2013-07-10 18:53:15: unlock_crshome=

      2013-07-10 18:53:15: user_is_superuser=1

      2013-07-10 18:53:15: ### Printing of configuration values complete ###

      2013-07-10 18:53:15: Oracle CRS stack is not configured yet

      2013-07-10 18:53:15: CRS is not yet configured. Hence, will proceed to configure CRS

      2013-07-10 18:53:15: Cluster-wide one-time actions... Done!

      2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home

      2013-07-10 18:53:15: Host name = rac2

      2013-07-10 18:53:15: CRS user = grid

      2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home

      2013-07-10 18:53:15: GPnP host = rac2

      2013-07-10 18:53:15: Oracle GPnP home = /u01/grid_home/gpnp

      2013-07-10 18:53:15: Oracle GPnP local home = /u01/grid_home/gpnp/rac2

      2013-07-10 18:53:15: GPnP directories verified.

      2013-07-10 18:53:15: Checking to see if Oracle CRS stack is already configured

      2013-07-10 18:53:15: Oracle CRS stack is not configured yet

      2013-07-10 18:53:15: ---Checking local gpnp setup...

      2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" does not exist

      2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" does not exist

      2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" does not exist

      2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0

      2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: INVALID (bad profile/wallet)

      2013-07-10 18:53:15: ---Checking cluster-wide gpnp setup...

      2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1

      2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer

      2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer

      2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer '

      2013-07-10 18:53:15: Removing file /tmp/file0qKE0c

      2013-07-10 18:53:15: Successfully removed file: /tmp/file0qKE0c

      2013-07-10 18:53:15: /bin/su successfully executed

       

       

      2013-07-10 18:53:15: gpnptool: rc=0

      2013-07-10 18:53:15: gpnptool output:

      Profile signature is valid.

       

       

      2013-07-10 18:53:15: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/peer"

      2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer

      2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer

      2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer '

      2013-07-10 18:53:16: Removing file /tmp/filebkOtBv

      2013-07-10 18:53:16: Successfully removed file: /tmp/filebkOtBv

      2013-07-10 18:53:16: /bin/su successfully executed

       

       

      2013-07-10 18:53:16: gpnptool: rc=0

      2013-07-10 18:53:16: gpnptool output:

      Profile signature is valid.

       

       

      2013-07-10 18:53:16: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/prdr"

      2013-07-10 18:53:16: chk gpnphome /u01/grid_home/gpnp: OK

      2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.

      2013-07-10 18:53:16: gpnp setup checked: local valid? 0 cluster-wide valid? 1

      2013-07-10 18:53:16: Taking cluster-wide setup as local

      2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/profiles/peer/profile.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml"

      2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" => (grid,oinstall)

      2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/peer/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso"

      2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" => (grid,oinstall)

      2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/prdr/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso"

      2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" => (grid,oinstall)

      2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/profiles/peer/profile_orig.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml"

      2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml" => (grid,oinstall)

      2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/root/ewallet.p12" => "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12"

      2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12" => (grid,oinstall)

      2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/pa/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso"

      2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso" => (grid,oinstall)

      2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/root/b64certificate.txt" => "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt"

      2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt" => (grid,oinstall)

      2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/peer/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt"

      2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt" => (grid,oinstall)

      2013-07-10 18:53:16:   copy "/u01/grid_home/gpnp/wallets/pa/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt"

      2013-07-10 18:53:16:   set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt" => (grid,oinstall)

      2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.

      2013-07-10 18:53:16: gpnp setup: GOTCLUSTERWIDE

      2013-07-10 18:53:16: Validating for SI-CSS configuration

      2013-07-10 18:53:16: Retrieving OCR main disk location

      2013-07-10 18:53:16: Opening file OCRCONFIG

      2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc

      2013-07-10 18:53:16: Unable to retrieve ocr disk info

      2013-07-10 18:53:16: Checking to see if any 9i GSD is up

      2013-07-10 18:53:16: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so

      2013-07-10 18:53:16: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so

      2013-07-10 18:53:16: SKGXN library file does not exists

      2013-07-10 18:53:16: OLR location = /u01/grid_home/cdata/rac2.olr

      2013-07-10 18:53:16: Oracle CRS Home = /u01/grid_home

      2013-07-10 18:53:16: Validating /etc/oracle/olr.loc file for OLR location /u01/grid_home/cdata/rac2.olr

      2013-07-10 18:53:16: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig

      2013-07-10 18:53:16: Oracle CRS home = /u01/grid_home

      2013-07-10 18:53:16: Oracle cluster name = rac-scan

      2013-07-10 18:53:16: OCR locations = +CRS

      2013-07-10 18:53:16: Validating OCR

      2013-07-10 18:53:16: Retrieving OCR location used by previous installations

      2013-07-10 18:53:16: Opening file OCRCONFIG

      2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc

      2013-07-10 18:53:16: Opening file OCRCONFIG

      2013-07-10 18:53:16: Value () is set for key=ocrmirrorconfig_loc

      2013-07-10 18:53:16: Opening file OCRCONFIG

      2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc3

      2013-07-10 18:53:16: Opening file OCRCONFIG

      2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc4

      2013-07-10 18:53:16: Opening file OCRCONFIG

      2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc5

      2013-07-10 18:53:16: Checking if OCR sync file exists

      2013-07-10 18:53:16: No need to sync OCR file

      2013-07-10 18:53:16: OCR_LOCATION=+CRS

      2013-07-10 18:53:16: OCR_MIRROR_LOCATION=

      2013-07-10 18:53:16: OCR_MIRROR_LOC3=

      2013-07-10 18:53:16: OCR_MIRROR_LOC4=

      2013-07-10 18:53:16: OCR_MIRROR_LOC5=

      2013-07-10 18:53:16: Current OCR location=

      2013-07-10 18:53:16: Current OCR mirror location=

      2013-07-10 18:53:16: Current OCR mirror loc3=

      2013-07-10 18:53:16: Current OCR mirror loc4=

      2013-07-10 18:53:16: Current OCR mirror loc5=

      2013-07-10 18:53:16: Verifying current OCR settings with user entered values

      2013-07-10 18:53:16: Setting OCR locations in /etc/oracle/ocr.loc

      2013-07-10 18:53:16: Validating OCR locations in /etc/oracle/ocr.loc

      2013-07-10 18:53:16: Checking for existence of /etc/oracle/ocr.loc

      2013-07-10 18:53:16: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig

      2013-07-10 18:53:16: Setting ocr location +CRS

      2013-07-10 18:53:16: Creating or upgrading Oracle Local Registry (OLR)

      2013-07-10 18:53:16: OLR successfully created or upgraded

      2013-07-10 18:53:16: /u01/grid_home/bin/clscfg -localadd

      2013-07-10 18:53:16: Keys created in the OLR successfully

      2013-07-10 18:53:16: GPnP setup state: new-cluster-wide

      2013-07-10 18:53:16: GPnP cluster configuration already performed

      2013-07-10 18:53:16: Registering ohasd

      2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/init.ohasd

      2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/init.ohasd to /etc/init.d directory

      2013-07-10 18:53:16: Setting init.ohasd permission in /etc/init.d directory

      2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/ohasd

      2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/ohasd to /etc/init.d directory

      2013-07-10 18:53:16: Setting ohasd permission in /etc/init.d directory

      2013-07-10 18:53:16: Removing "/etc/rc.d/rc3.d/S96ohasd"

      2013-07-10 18:53:16: Removing file /etc/rc.d/rc3.d/S96ohasd

      2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd

      2013-07-10 18:53:16: Failed to remove file:

      2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd

      2013-07-10 18:53:16: Removing "/etc/rc.d/rc5.d/S96ohasd"

      2013-07-10 18:53:16: Removing file /etc/rc.d/rc5.d/S96ohasd

      2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd

      2013-07-10 18:53:16: Failed to remove file:

      2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd

      2013-07-10 18:53:16: Removing "/etc/rc.d/rc0.d/K19ohasd"

      2013-07-10 18:53:16: Removing file /etc/rc.d/rc0.d/K19ohasd

      2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd

      2013-07-10 18:53:16: Failed to remove file:

      2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd

      2013-07-10 18:53:16: Removing "/etc/rc.d/rc1.d/K19ohasd"

      2013-07-10 18:53:16: Removing file /etc/rc.d/rc1.d/K19ohasd

      2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd

      2013-07-10 18:53:16: Failed to remove file:

      2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd

      2013-07-10 18:53:16: Removing "/etc/rc.d/rc2.d/K19ohasd"

      2013-07-10 18:53:16: Removing file /etc/rc.d/rc2.d/K19ohasd

      2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd

      2013-07-10 18:53:16: Failed to remove file:

      2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd

      2013-07-10 18:53:16: Removing "/etc/rc.d/rc4.d/K19ohasd"

      2013-07-10 18:53:16: Removing file /etc/rc.d/rc4.d/K19ohasd

      2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd

      2013-07-10 18:53:16: Failed to remove file:

      2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd

      2013-07-10 18:53:16: Removing "/etc/rc.d/rc6.d/K19ohasd"

      2013-07-10 18:53:16: Removing file /etc/rc.d/rc6.d/K19ohasd

      2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd

      2013-07-10 18:53:16: Failed to remove file:

      2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd

      2013-07-10 18:53:16: The file ohasd has been successfully linked to the RC directories

      2013-07-10 18:53:16: Starting ohasd

      2013-07-10 18:53:16: itab entries=

      2013-07-10 18:53:21: Created backup /etc/inittab.no_crs

      2013-07-10 18:53:21: Appending to /etc/inittab.tmp:

      2013-07-10 18:53:21: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

       

       

      2013-07-10 18:53:21: Done updating /etc/inittab.tmp

      2013-07-10 18:53:21: Saved /etc/inittab.crs

      2013-07-10 18:53:21: Installed new /etc/inittab

      2013-07-10 18:53:36: ohasd is starting

      2013-07-10 18:53:36: Checking ohasd

      2013-07-10 18:53:37: ohasd started successfully

      2013-07-10 18:53:37: Creating CRS resources and dependencies

      2013-07-10 18:53:37: Configuring HASD

      2013-07-10 18:53:37: Registering type ora.daemon.type

      2013-07-10 18:53:37: Registering type ora.mdns.type

      2013-07-10 18:53:37: Registering type ora.gpnp.type

      2013-07-10 18:53:38: Registering type ora.gipc.type

      2013-07-10 18:53:38: Registering type ora.cssd.type

      2013-07-10 18:53:38: Registering type ora.cssdmonitor.type

      2013-07-10 18:53:39: Registering type ora.crs.type

      2013-07-10 18:53:39: Registering type ora.evm.type

      2013-07-10 18:53:39: Registering type ora.ctss.type

      2013-07-10 18:53:40: Registering type ora.asm.type

      2013-07-10 18:53:40: Registering type ora.drivers.acfs.type

      2013-07-10 18:53:40: Registering type ora.diskmon.type

      2013-07-10 18:53:51: ADVM/ACFS is configured

       

       

      2013-07-10 18:53:51: Successfully created CRS resources for cluster daemon and ASM

      2013-07-10 18:53:51: Checking if initial configuration has been performed

      2013-07-10 18:53:51: Starting CSS in exclusive mode

      2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'

      2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'

      2013-07-10 18:54:19: CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded

      2013-07-10 18:54:19: CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded

      2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'

      2013-07-10 18:54:19: CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded

      2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'

      2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded

      2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssd' on 'rac2'

      2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'

      2013-07-10 18:54:19: CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded

      2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded

      2013-07-10 18:54:19: Querying for existing CSS voting disks

      2013-07-10 18:54:19: Performing initial configuration for cluster

      2013-07-10 18:54:21: Start of resource "ora.ctssd -init" Succeeded

      2013-07-10 18:54:21: Configuring ASM via ASMCA

      2013-07-10 18:54:21: Executing as grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM

      2013-07-10 18:54:21: Running as user grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM

      2013-07-10 18:54:21:   Invoking "/u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM" as user "grid"

      2013-07-10 18:54:40: Configuration of ASM failed, see logs for details

      2013-07-10 18:54:40: Did not succssfully configure and start ASM

      2013-07-10 18:54:40: Exiting exclusive mode

      2013-07-10 18:54:40: Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init

      2013-07-10 18:54:40: Stop of resource "ora.crsd -init" failed

      2013-07-10 18:54:40: Failed to stop CRSD

      2013-07-10 18:55:04: Initial cluster configuration failed.  See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details

       

      Also below are some of the configs related to rac2 node

       

      [root@rac2 rac2]# rpm -qa | grep oracleasm

       

       

      oracleasmlib-2.0.4-1.el5

      oracleasm-support-2.1.8-1.el5

      oracleasm-2.6.18-274.el5xen-2.0.5-1.el5

      oracleasm-2.6.18-274.el5-2.0.5-1.el5

      oracleasm-2.6.18-274.el5debug-2.0.5-1.el5

      oracleasm-2.6.18-274.el5-debuginfo-2.0.5-1.el5

       

       

       

       

      [root@rac2 rac2]# /usr/sbin/oracleasm configure

      ORACLEASM_ENABLED=true

      ORACLEASM_UID=grid

      ORACLEASM_GID=asmadmin

      ORACLEASM_SCANBOOT=true

      ORACLEASM_SCANORDER=""

      ORACLEASM_SCANEXCLUDE=""

      ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"

       

       

       

       

      [root@rac2 rac2]# /usr/sbin/oracleasm status

      Checking if ASM is loaded: yes

      Checking if /dev/oracleasm is mounted: yes

       

       

       

       

      [root@rac2 rac2]# /usr/sbin/oracleasm listdisks

      CRS1

      DATA1

      FRA1

       

      [root@rac2 rac2]# ls -l /dev/oracleasm/disks/

      total 0

      brw-rw---- 1 grid asmadmin 8, 17 Jul 10 18:35 CRS1

      brw-rw---- 1 grid asmadmin 8, 33 Jul 10 18:36 DATA1

      brw-rw---- 1 grid asmadmin 8, 49 Jul 10 18:36 FRA1

       

       

      [root@rac2 rac2]# cat /etc/hosts

      # Do not remove the following line, or various programs

      # that require network functionality will fail.

      127.0.0.1               localhost.localdomain localhost

      ::1             localhost6.localdomain6 localhost6

       

       

      #Public IP's(eth0)

      192.168.0.101    rac1.naveed.com    rac1

      192.168.0.102    rac2.naveed.com    rac2

       

       

      #Private IP's(eth1)

      192.168.1.101    rac1-prv.naveed.com   rac1-prv

      192.168.1.102    rac2-prv.naveed.com   rac2-prv

       

       

      #VIPS

      192.168.0.221    rac1-vip.naveed.com   rac1-vip

      192.168.0.222    rac2-vip.naveed.com   rac2-vip

       

       

      #DNS server IP

      192.168.0.10    naveeddns.naveed.com   naveeddns

       

       

      [root@rac2 rac2]#

       

      Thanks in advance

        • 1. Re: root.ah fails on 2nd node(rac2) with [ ORA-15018,ORA-15017,ORA-15003 ]
          rarain

          Hi,

           

          Deinstall everything and start a fresh CRS installation but before to proceed for installation make sure that both the node have correct disks permission and better to format disk using DD command.

           

          Thanks

          • 2. Re: root.ah fails on 2nd node(rac2) with [ ORA-15018,ORA-15017,ORA-15003 ]
            993159

            Hi,

             

            First of all thanks a lot for the response. You wont't beleive this is my 7th fresh installation and everytime in node 2 i m hit with this same error.

             

             

            Also i tried below procedure instead of fresh installation

             

            once i deconfig & rerun (./rootcrs.pl -verbose -deconfig -force) on node 2

            Using configuration parameter file: ./crsconfig_params

            PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type

            PRCR-1068 : Failed to query resources

            Cannot communicate with crsd

            PRCR-1070 : Failed to check if resource ora.gsd is registered

            Cannot communicate with crsd

            PRCR-1070 : Failed to check if resource ora.ons is registered

            Cannot communicate with crsd

             

             

            CRS-4535: Cannot communicate with Cluster Ready Services

            CRS-4000: Command Stop failed, or completed with errors.

            CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'

            CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac2'

            CRS-2673: Attempting to stop 'ora.asm' on 'rac2'

            CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'

            CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'

            CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded

            CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded

            CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'

            CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded

            CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded

            CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'

            CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded

            CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'

            CRS-2677: Stop of 'ora.drivers.acfs' on 'rac2' succeeded

            CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded

            CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'

            CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded

            CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed

            CRS-4133: Oracle High Availability Services has been stopped.

            Successfully deconfigured Oracle clusterware stack on this node

             

             




            [root@rac2 grid_home]# ./root.sh

            Performing root user operation for Oracle 11g

             

             

            The following environment variables are set as:

                ORACLE_OWNER= grid

                ORACLE_HOME=  /u01/grid_home

             

             

            Enter the full pathname of the local bin directory: [/usr/local/bin]:

            The contents of "dbhome" have not changed. No need to overwrite.

            The contents of "oraenv" have not changed. No need to overwrite.

            The contents of "coraenv" have not changed. No need to overwrite.

             

             

            Entries will be added to the /etc/oratab file as needed by

            Database Configuration Assistant when a database is created

            Finished running generic part of root script.

            Now product-specific root actions will be performed.

            Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params

            User ignored Prerequisites during installation

            OLR initialization - successful

            Adding Clusterware entries to inittab

            CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'

            CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded

            CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'

            CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded

            CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'

            CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'

            CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded

            CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded

            CRS-2672: Attempting to start 'ora.cssd' on 'rac2'

            CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'

            CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded

            CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded

             

             

            ASM created and started successfully.

             

             

            Disk Group CRS mounted successfully.

             

             

            clscfg: -install mode specified

            Successfully accumulated necessary OCR keys.

            Creating OCR keys for user 'root', privgrp 'root'..

            Operation successful.

            Successful addition of voting disk 636af26485ef4f27bfec31523aaa0660.

            Successfully replaced voting disk group with +CRS.

            CRS-4266: Voting file(s) successfully replaced

            ##  STATE    File Universal Id                File Name Disk group

            --  -----    -----------------                --------- ---------

            1. ONLINE   636af26485ef4f27bfec31523aaa0660 (ORCL:CRS1) [CRS]

            Located 1 voting disk(s).

            Start of resource "ora.crsd" failed

            CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'rac2'

            CRS-4000: Command Start failed, or completed with errors.

            Failed to start Oracle Grid Infrastructure stack

            Failed to start Cluster Ready Services at /u01/grid_home/crs/install/crsconfig_lib.pm line 1286.

            /u01/grid_home/perl/bin/perl -I/u01/grid_home/perl/lib -I/u01/grid_home/crs/install /u01/grid_home/crs/install/rootcrs.pl execution failed


            • 3. Re: root.ah fails on 2nd node(rac2) with [ ORA-15018,ORA-15017,ORA-15003 ]
              saurabh

              post the reference link you are using for installing. So we can check in it that is there any step missing or something.

              • 4. Re: root.ah fails on 2nd node(rac2) with [ ORA-15018,ORA-15017,ORA-15003 ]
                993159

                Hi All,

                 

                I was installing Clusterware using ASM on VMware shared disks.  When I created the independent persistent virtual disks, I left the “allocate all disk space now” option unselected. Oracleasm was happy on both RAC nodes. The Oracle installer was happy when it created +ASM1 on the first RAC node. But when the ASM instance started on node 2 it did not like the “virtually provisioned” disk. The +ASM2 instance was not open and was complaining about one of the shared disks being corrupt at a certain byte. When I checked the virtual disk files, I saw that Oracle was trying to read past the end of the file. I started all over with new fully allocated shared disk and that fixed the problem. Everything is up and running now.