This discussion is archived
13 Replies Latest reply: Feb 6, 2013 6:47 AM by user13487657 RSS

root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource

475068 Newbie
Currently Being Moderated
Dear all,

I am trying to install clean Oracle 11.2.0.3 grid infrastructure on a two node cluster running on Solaris 5.10.

- Cluster verification was successfully on both nodes; No warning or issues;
- I am using 2 network cards for the public and 2 for the private interconnect;
- OCR is stored on ASM
- Firewall is disabled on both nodes
- SCAN is being configured on the DNS (not added in /etc/hosts)
- GNS is not used
- hosts file is identical (except the primary hostname)

The problem: root.sh fails on the 2nd (remote) node, because it fails to start the "ora.asm" resource. However, the root.sh has completed successfully on the 1st node.. Somehow, root.sh doesn't create +ASM2 instance on the remote (host2) node.

root.sh was executed first on the local node (host1) and after the successful execution was started on the remote (host2) node.

Output from host1 (working):
===================
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'host1'
CRS-2676: Start of 'ora.mdnsd' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'host1'
CRS-2676: Start of 'ora.gpnpd' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'host1'
CRS-2672: Attempting to start 'ora.gipcd' on 'host1'
CRS-2676: Start of 'ora.cssdmonitor' on 'host1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'host1'
CRS-2672: Attempting to start 'ora.diskmon' on 'host1'
CRS-2676: Start of 'ora.diskmon' on 'host1' succeeded
CRS-2676: Start of 'ora.cssd' on 'host1' succeeded

ASM created and started successfully.

Disk Group CRS created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 4373be34efab4f01bf79f6c5362acfd3.
Successful addition of voting disk 7fd725fa4d904f07bf76cecf96791547.
Successful addition of voting disk a9c85297bdd74f3abfd86899205aaf17.
Successfully replaced voting disk group with +CRS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 4373be34efab4f01bf79f6c5362acfd3 (/dev/rdsk/c4t600A0B80006E2CC40000C6674E82AA57d0s4) [CRS]
2. ONLINE 7fd725fa4d904f07bf76cecf96791547 (/dev/rdsk/c4t600A0B80006E2CC40000C6694E82AADDd0s4) [CRS]
3. ONLINE a9c85297bdd74f3abfd86899205aaf17 (/dev/rdsk/c4t600A0B80006E2F100000C7744E82AC7Ad0s4) [CRS]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'host1'
CRS-2676: Start of 'ora.asm' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.CRS.dg' on 'host1'
CRS-2676: Start of 'ora.CRS.dg' on 'host1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'host1'
CRS-2676: Start of 'ora.registry.acfs' on 'host1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Name Type Target State Host
------------------------------------------------------------
ora.CRS.dg ora....up.type ONLINE ONLINE host1
ora....ER.lsnr ora....er.type ONLINE ONLINE host1
ora....N1.lsnr ora....er.type ONLINE ONLINE host1
ora....N2.lsnr ora....er.type ONLINE ONLINE host1
ora....N3.lsnr ora....er.type ONLINE ONLINE host1
ora.asm ora.asm.type ONLINE ONLINE host1
ora....SM1.asm application ONLINE ONLINE host1
ora....B1.lsnr application ONLINE ONLINE host1
ora....db1.gsd application OFFLINE OFFLINE
ora....db1.ons application ONLINE ONLINE host1
ora....db1.vip ora....t1.type ONLINE ONLINE host1
ora.cvu ora.cvu.type ONLINE ONLINE host1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE host1
ora.oc4j ora.oc4j.type ONLINE ONLINE host1
ora.ons ora.ons.type ONLINE ONLINE host1
ora....ry.acfs ora....fs.type ONLINE ONLINE host1
ora.scan1.vip ora....ip.type ONLINE ONLINE host1
ora.scan2.vip ora....ip.type ONLINE ONLINE host1
ora.scan3.vip ora....ip.type ONLINE ONLINE host1

Output from host2 (failing):
===================
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node billdb1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster

Start of resource "ora.asm" failed
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'host2'
CRS-2676: Start of 'ora.drivers.acfs' on 'host2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'host2'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
*. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log".*
CRS-2674: Start of 'ora.asm' on 'host2' failed
CRS-2679: Attempting to clean 'ora.asm' on 'host2'
CRS-2681: Clean of 'ora.asm' on 'host2' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'host2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'host2' succeeded
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Grid Infrastructure stack
Failed to start ASM at /u01/11.2.0/grid/crs/install/crsconfig_lib.pm line 1272.
/u01/11.2.0/grid/perl/bin/perl -I/u01/11.2.0/grid/perl/lib -I/u01/11.2.0/grid/crs/install /u01/11.2.0/grid/crs/install/rootcrs.pl execution failed

Contents of "/u01/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_host2.log"
=============================================
CRS-2672: Attempting to start 'ora.asm' on 'host2'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log".
CRS-2674: Start of 'ora.asm' on 'host2' failed
CRS-2679: Attempting to clean 'ora.asm' on 'host2'
CRS-2681: Clean of 'ora.asm' on 'host2' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'host2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'host2' succeeded
CRS-4000: Command Start failed, or completed with errors.
2011-10-24 19:36:54: Failed to start Oracle Grid Infrastructure stack
2011-10-24 19:36:54: ###### Begin DIE Stack Trace ######
2011-10-24 19:36:54: Package File Line Calling
2011-10-24 19:36:54: --------------- -------------------- ---- ----------
2011-10-24 19:36:54: 1: main rootcrs.pl 375 crsconfig_lib::dietrap
2011-10-24 19:36:54: 2: crsconfig_lib crsconfig_lib.pm 1272 main::__ANON__
2011-10-24 19:36:54: 3: crsconfig_lib crsconfig_lib.pm 1171 crsconfig_lib::start_cluster
2011-10-24 19:36:54: 4: main rootcrs.pl 803 crsconfig_lib::perform_start_cluster
2011-10-24 19:36:54: ####### End DIE Stack Trace #######

Shortened output from "/u01/11.2.0/grid/log/host2/agent/ohasd/oraagent_grid/oraagent_grid.log"

2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] clean {
2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] InstAgent::stop_option stop mode immediate option 1
2011-10-24 19:35:48.726: [ora.asm][9] {0:0:224} [start] InstAgent::stop {
2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] InstAgent::stop original reason system do shutdown abort
2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] ConnectionPool::resetConnection s_statusOfConnectionMap 00ab1948
2011-10-24 19:35:48.727: [ora.asm][9] {0:0:224} [start] ConnectionPool::resetConnection sid +ASM2 status  2
2011-10-24 19:35:48.728: [ora.asm][9] {0:0:224} [start] Gimh::check OH /u01/11.2.0/grid SID +ASM2
2011-10-24 19:35:48.728: [ora.asm][9] {0:0:224} [start] Gimh::check condition changes to (GIMH_NEXT_NUM) 0,1,7 exists
2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] (:CLSN00006:)AsmAgent::check failed gimh state 0
2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 0
2011-10-24 19:35:48.729: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet { entry
2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet procr_get_conf: retval [0] configured [1] local only [0] error buffer []
2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet procr_get_conf: OCR loc [0], Disk Group : [+CRS]
2011-10-24 19:35:48.730: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet m_ocrDgpSet 015fba90 dgName CRS
2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet ocrret 0 found 1
2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet ocrDgpSet CRS
2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::initOcrDgpSet exit }
2011-10-24 19:35:48.731: [ora.asm][9] {0:0:224} [start] DgpAgent::ocrDgCheck Entry {
2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] DgpAgent::getConnxn new pool
2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] DgpAgent::getConnxn new pool m_oracleHome:/u01/11.2.0/grid m_oracleSid:+ASM2 m_usrOraEnv:
2011-10-24 19:35:48.732: [ora.asm][9] {0:0:224} [start] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/11.2.0/grid, m_oracleSid:+ASM2, m_usrOraEnv:
2011-10-24 19:35:48.733: [ora.asm][9] {0:0:224} [start] ConnectionPool::addConnection m_oracleHome:/u01/11.2.0/grid m_oracleSid:+ASM2 m_usrOraEnv: pConnxn:
01fcdf10
2011-10-24 19:35:48.733: [ora.asm][9] {0:0:224} [start] Utils::getCrsHome crsHome /u01/11.2.0/grid


2011-10-24 19:35:51.969: [ora.asm][14] {0:0:224} [check] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/11.2.0/grid/bin/oracle)(ARGV0=o
racle+ASM2)(ENVS='ORACLE_HOME=/u01/11.2.0/grid,ORACLE_SID=+ASM2')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM2)))
2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] ConnectionPool::getConnection 260 pConnxn 013e40a0
2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] DgpAgent::getConnxn connected
2011-10-24 19:35:51.971: [ora.asm][14] {0:0:224} [check] InstConnection::connectInt: server not attached
2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
SVR4 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0

2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] InstConnection::connectInt (2) Exception OCIException
2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] InstConnection:connect:excp OCIException OCI error 1034
2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus excp ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
SVR4 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0

2011-10-24 19:35:52.190: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus asm inst is down or going down
2011-10-24 19:35:52.191: [ora.asm][14] {0:0:224} [check] DgpAgent::queryDgStatus dgName CRS ret 1
2011-10-24 19:35:52.191: [ora.asm][14] {0:0:224} [check] (:CLSN00100:)DgpAgent::ocrDgCheck OCR dgName CRS state 1
2011-10-24 19:35:52.192: [ora.asm][14] {0:0:224} [check] ConnectionPool::releaseConnection InstConnection 013e40a0
2011-10-24 19:35:52.192: [ora.asm][14] {0:0:224} [check] AsmAgent::check ocrCheck 2 m_OcrOnline 0 m_OcrTimer 0
2011-10-24 19:35:52.193: [ora.asm][14] {0:0:224} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 32 useFilter 0
2011-10-24 19:35:52.197: [ COMMCRS][23]clsc_connect: (1020d39d0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

Please advice for any workaround or a metalink note.

Thanks in advance!
  • 1. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    696867 Explorer
    Currently Being Moderated
    I would start looking into:
    Is there anything in the ASM alert log on the failed node?
    Is the shared storage accessible from the failed node (i.e, now after the failure)?
    Is there anything in the system logs around the this error?


    HTH
  • 2. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    475068 Newbie
    Currently Being Moderated
    Thanks for the fast reply!

    - Yes, the shared storage is accessible.
    - The alert log for the +ASM2 clearly shows that ASM instance has started normally using default parameters and at one point PMON process dumped.
    - The system logs just shows that there is an error executing "crswrapexece.pl"


    System Log
    ===================
    *Oct 24 19:25:03 host2 root: [ID 702911 user.error] exec /u01/11.2.0/grid/perl/bin/perl -I/u01/11.2.0/grid/perl/lib /u01/11.2.0/grid/bin/crswrapexece.pl /*
    u01/11.2.0/grid/crs/install/s_crsconfig_host2_env.txt /u01/11.2.0/grid/bin/ohasd.bin "reboot"
    Oct 24 19:26:33 host2 oracleoks: [ID 902884 kern.notice] [Oracle OKS] mallocing log buffer, size=10485760
    Oct 24 19:26:33 host2 oracleoks: [ID 714332 kern.notice] [Oracle OKS] log buffer = 0x301780fcb50, size 10485760
    Oct 24 19:26:33 host2 oracleoks: [ID 400061 kern.notice] NOTICE: [Oracle OKS] ODLM hash size 16384
    Oct 24 19:26:33 host2 oracleoks: [ID 160659 kern.notice] NOTICE: OKSK-00004: Module load succeeded. Build information: (LOW DEBUG) USM_11.2.0.3.0_SOLAR
    IS.SPARC64_110803.1 2011/08/11 02:38:30
    Oct 24 19:26:33 host2 pseudo: [ID 129642 kern.info] pseudo-device: oracleadvm0
    Oct 24 19:26:33 host2 genunix: [ID 936769 kern.info] oracleadvm0 is /pseudo/oracleadvm@0
    Oct 24 19:26:33 host2 oracleoks: [ID 141287 kern.notice] NOTICE: ADVMK-00001: Module load succeeded. Build information: (LOW DEBUG) - USM_11.2.0.3.0_SOL
    ARIS.SPARC64_110803.1 built on 2011/08/11 02:40:17.
    Oct 24 19:26:33 host2 oracleacfs: [ID 202941 kern.notice] NOTICE: [Oracle ACFS] FCB hash size 16384
    Oct 24 19:26:33 host2 oracleacfs: [ID 671725 kern.notice] NOTICE: [Oracle ACFS] buffer cache size 511MB (79884 buckets)
    Oct 24 19:26:33 host2 oracleacfs: [ID 730054 kern.notice] NOTICE: [Oracle ACFS] DLM hash size 16384
    Oct 24 19:26:33 host2 oracleoks: [ID 617314 kern.notice] NOTICE: ACFSK-0037: Module load succeeded. Build information: (LOW DEBUG) USM_11.2.0.3.0_SOLAR
    IS.SPARC64_110803.1 2011/08/11 02:42:45
    Oct 24 19:26:33 host2 pseudo: [ID 129642 kern.info] pseudo-device: oracleacfs0
    Oct 24 19:26:33 host2 genunix: [ID 936769 kern.info] oracleacfs0 is /pseudo/oracleacfs@0
    Oct 24 19:26:36 host2 oracleoks: [ID 621795 kern.notice] NOTICE: OKSK-00010: Persistent OKS log opened at /u01/11.2.0/grid/log/host2/acfs/acfs.log.0.
    Oct 24 19:31:37 host2 last message repeated 1 time
    Oct 24 19:33:05 host2 CLSD: [ID 770310 daemon.notice] The clock on host host2 has been updated by the Cluster Time Synchronization Service to be synchr
    onous with the mean cluster time.

    ASM alert log
    ====================================================================
    <msg time='2011-10-24T19:35:48.776+01:00' org_id='oracle' comp_id='asm'
    client_id='' type='UNKNOWN' level='16'
    host_id='host2' host_addr='10.172.16.200' module=''
    pid='26406'>
    <txt>System state dump requested by (instance=2, osid=26396 (PMON)), summary=[abnormal instance termination].
    </txt>
    </msg>
    <msg time='2011-10-24T19:35:48.778+01:00' org_id='oracle' comp_id='asm'
    client_id='' type='UNKNOWN' level='16'
    host_id='host2' host_addr='10.172.16.200' module=''
    pid='26406'>
    <txt>System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_26406.trc
    </txt>
    </msg>
    <msg time='2011-10-24T19:35:48.927+01:00' org_id='oracle' comp_id='asm'
    type='UNKNOWN' level='16' host_id='host2'
    host_addr='10.172.16.200' pid='26470'>
    <txt>ORA-1092 : opitsk aborting process
    </txt>
    </msg>
    <msg time='2011-10-24T19:35:49.128+01:00' org_id='oracle' comp_id='asm'
    type='UNKNOWN' level='16' host_id='host2'
    host_addr='10.172.16.200' pid='26472'>
    <txt>ORA-1092 : opitsk aborting process
    </txt>
    </msg>

    Output from "/u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_26406.trc"

    REQUEST:system state dump at level 10, requested by (instance=2, osid=26396 (PMON)), summary=[abnormal instance termination].
    kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).
    ===================================================
    SYSTEM STATE (level=10)

    Orapids on dead process list: [count = 0]
    PROCESS 1:
    ----------------------------------------
    SO: 0x3df098b50, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df098b50, name=process, file=ksu.h LINE:12616 ID:, pg=0
    (process) Oracle pid:1, ser:0, calls cur/top: 0x0/0x0
    flags : (0x20) PSEUDO
    flags2: (0x0), flags3: (0x10)
    intr error: 0, call error: 0, sess error: 0, txn error 0
    intr queue: empty
    ksudlp FALSE at location: 0
    (post info) last post received: 0 0 0
    last post received-location: No post
    last process to post me: none
    last post sent: 0 0 0
    last post sent-location: No post
    last process posted by me: none
    (latch info) wait_event=0 bits=0
    O/S info: user: , term: , ospid: (DEAD)
    OSD pid info: Unix process pid: 0, image: PSEUDO
    ----------------------------------------
    SO: 0x38000cef0, type: 5, owner: 0x3df098b50, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x0, name=kss parent, file=kss2.h LINE:138 ID:, pg=0
    PSO child state object changes :
    Dump of memory from 0x00000003DF722AC0 to 0x00000003DF722CC8
    3DF722AC0 00000000 00000000 00000000 00000000 [................]
    Repeat 31 times
    3DF722CC0 00000000 00000000 [........]
    PROCESS 2: PMON
    ----------------------------------------
    SO: 0x3df099bf8, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df099bf8, name=process, file=ksu.h LINE:12616 ID:, pg=0
    (process) Oracle pid:2, ser:1, calls cur/top: 0x3db6c8d30/0x3db6c8d30
    flags : (0xe) SYSTEM
    flags2: (0x0), flags3: (0x10)
    intr error: 0, call error: 0, sess error: 0, txn error 0
    intr queue: empty
    ksudlp FALSE at location: 0
    (post info) last post received: 0 0 136
    last post received-location: kjm.h LINE:1228 ID:kjmdmi: pmon to attach
    last process to post me: 3df0a2138 1 6
    last post sent: 0 0 137
    last post sent-location: kjm.h LINE:1230 ID:kjiath: pmon attached
    last process posted by me: 3df0a2138 1 6
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 0x3debbbf40
    O/S info: user: grid, term: UNKNOWN, ospid: 26396
    OSD pid info: Unix process pid: 26396, image: oracle@host2 (PMON)
    ----------------------------------------
    SO: 0x3d8800c18, type: 30, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df099bf8, name=ges process, file=kji.h LINE:3669 ID:, pg=0
    GES MSG BUFFERS: st=emp chunk=0x0 hdr=0x0 lnk=0x0 flags=0x0 inc=0
    outq=0 sndq=0 opid=0 prmb=0x0
    mbg=(0 0) mbg=(0 0) mbg[r]=(0 0)
    fmq=(0 0) fmq=(0 0) fmq[r]=(0 0)
    mop[s]=0 mop[q]=0 pendq=0 zmbq=0
    nonksxp_recvs=0
    ------------process 3d8800c18--------------------
    proc version : 0
    Local inst : 2
    pid : 26396
    lkp_inst : 2
    svr_mode : 0
    proc state : KJP_FROZEN
    Last drm hb acked : 0
    flags : x50
    ast_rcvd_svrmod : 0
    current lock op : 0
    Total accesses : 1
    Imm. accesses : 0
    Locks on ASTQ : 0
    Locks Pending AST : 0
    Granted locks : 0
    AST_Q:
    PENDING_Q:
    GRANTED_Q:
    ----------------------------------------
    SO: 0x3d9835198, type: 14, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
    proc=0x3df099bf8, name=channel handle, file=ksr2.h LINE:367 ID:, pg=0
    (broadcast handle) 3d9835198 flag: (2) ACTIVE SUBSCRIBER,
    owner: 3df099bf8 - ospid: 26396
    event: 1, last message event: 1,
    last message waited event: 1,
    next message: 0(0), messages read: 0
    channel: (3d9934df8) PMON actions channel [name: 2]
    scope: 7, event: 1, last mesage event: 0,
    publishers/subscribers: 0/1,
    messages published: 0
    heuristic msg queue length: 0
    ----------------------------------------
    SO: 0x3d9835008, type: 14, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
    proc=0x3df099bf8, name=channel handle, file=ksr2.h LINE:367 ID:, pg=0
    (broadcast handle) 3d9835008 flag: (2) ACTIVE SUBSCRIBER,
    owner: 3df099bf8 - ospid: 26396
    event: 1, last message event: 1,
    last message waited event: 1,
    next message: 0(0), messages read: 0
    channel: (3d9941e40) scumnt mount lock [name: 157]
    scope: 1, event: 12, last mesage event: 0,
    publishers/subscribers: 0/12,
    messages published: 0
    heuristic msg queue length: 0
    ----------------------------------------
    SO: 0x3de4a2b80, type: 4, owner: 0x3df099bf8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
    proc=0x3df099bf8, name=session, file=ksu.h LINE:12624 ID:, pg=0
    (session) sid: 33 ser: 1 trans: 0x0, creator: 0x3df099bf8
    flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
    flags2: (0x409) -/-/INC
    DID: , short-term DID:
    txn branch: 0x0
    oct: 0, prv: 0, sql: 0x0, psql: 0x0, user: 0/SYS
    ksuxds FALSE at location: 0
    service name: SYS$BACKGROUND
    Current Wait Stack:
    Not in wait; last wait ended 0.666415 sec ago
    Wait State:
    fixed_waits=0 flags=0x21 boundary=0x0/-1
    Session Wait History:
    elapsed time of 0.666593 sec since last wait
    0: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=63 seq_num=64 snap_id=1
    wait times: snap=3.000089 sec, exc=3.000089 sec, total=3.000089 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.002067 sec of elapsed time
    1: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=62 seq_num=63 snap_id=1
    wait times: snap=3.010111 sec, exc=3.010111 sec, total=3.010111 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.001926 sec of elapsed time
    2: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=61 seq_num=62 snap_id=1
    wait times: snap=3.125286 sec, exc=3.125286 sec, total=3.125286 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.003361 sec of elapsed time
    3: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
    wait_id=60 seq_num=61 snap_id=1
    wait times: snap=3.000081 sec, exc=3.000081 sec, total=3.000081 sec
    wait times: max=3.000000 sec
    wait counts: calls=1 os=1
    occurred after 0.002102 sec of elapsed time
    4: waited for 'pmon timer'
    duration=0x12c, =0x0, =0x0
  • 3. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    475068 Newbie
    Currently Being Moderated
    I think I found a relevant metalink note:

    https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=PROBLEM&id=1346332.1

    and my alert.log is flooded with

    kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).
  • 4. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    884376 Newbie
    Currently Being Moderated
    I'm running into the exact same problem. Following the metalink article didnt help. I thouhgt it would as the primary node picked up the wrong network interface for cluster interconnect, however, even after restarting the primary node (and it picked up the correct vnet) I've had no luck.

    This is a clean solaris Sparc 11.2.0.3 install inside LDOMs.

    No GNS. /etc/hosts are clean.
    Shared disks are accessible. However, the cluvfy said, it wasnt sure. I'm not sure why?!

    11.2.0.2 installed like a charm, and this is the same hardware with fresh OS installs and disks .

    The first node installed perfectly.
  • 5. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    884376 Newbie
    Currently Being Moderated
    Quick Update.

    I found that the MTU on the plumbed nic's (by the cluster software) was wrong. I used Ops Center to create the Control domain on this blade, and it had incorrectly set the MTU to 9126 instead of 1500.

    Correcting the MTU and restarting the root.sh worked like a charm.
  • 6. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    475068 Newbie
    Currently Being Moderated
    Well after following the metalink note, indeed the issue was with the HAIP and the different IP/subnet, but upon restarting the 1st node it did picked up the appropriate IPs and than I've managed to start the ora.asm resource manually on the 2nd node; But the cluster started to reallocate all the resources to the 2nd node and shutting down crs resources on the 1st node; And similar, once I start the crs resources on the 1st node, the 2nd was shutting down;

    Disks used for the CRS are available to both nodes; We have checked God knows how many times the privileges of the disk devices used, did many cluster verification tests and everything is clean.

    I've raised an SR to Oracle like 10 days ago and we are still working on it. I will update this thread with the outcome of the SR.
  • 7. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    475068 Newbie
    Currently Being Moderated
    Thanks, I will try that one tomorrow; Ours is 1500 MTU; I wonder why Oracle Support didn't comment on that, since the output is clearly visible in the cluster verification ....
  • 8. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    884376 Newbie
    Currently Being Moderated
    I think the Db guys are far apart from the OS guys. Does your cluvfy show 1500 as well? Thats how I found that the plumbed nic's were the wrong MTU.
  • 9. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    884376 Newbie
    Currently Being Moderated
    Another update:

    I found that the disks didnt have the permission 0660. Also - make sure the disks are owned by oinstall.
  • 10. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    475068 Newbie
    Currently Being Moderated
    Sorry for not replying sooner but the OTN was down, at least not reachable from Europe. Disk permissions are fine, we will try with the MTU that you suggest and see what happens.

    Thanks!
  • 11. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    884376 Newbie
    Currently Being Moderated
    Just curious...did the MTU setting fix it?
  • 12. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    980037 Newbie
    Currently Being Moderated
    I have same exact errors and root.sh failed on second node
    MTU didn't fix the issue in my case, it was 9000 on private connect I changed it to 1500 restarted network and still didn't work
    Also the link for metalink document doesn't work
    I am stuck with the root.sh failing on second node and the oui is waiting to finish configuration
    Any solution out there? is there any harm of keep trying to re run root.sh on 2nd node?
    can I let the installer continue and finish installation despite the fact that root.sh failed on node 2?
  • 13. Re: root.sh fails on 11.2.0.3 clusterware while starting 'ora.asm' resource
    user13487657 Newbie
    Currently Being Moderated
    we have similar issue.

    deconfig root.sh

    reboot server

    ans run root.sh again

    this may help

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points