This discussion is archived
6 Replies Latest reply: Oct 24, 2012 2:34 AM by AlleT RSS

unable to start crs

AlleT Explorer
Currently Being Moderated
Hi,

I installed grid infrastructure 11.2.0.3 on a single RHEL 5.5 node.

I put OCR and voting disks in an ASM disk group.

Everything worked fine but when I rebooted the machine CRS didn't start up automatically.

The logs show crs is missing OCR and voting disks because they are on ASM:

2012-10-22 18:54:22.249
[ohasd(9097)]CRS-2112:The OLR service started on node ludo.
2012-10-22 18:54:22.345
[ohasd(9097)]CRS-1301:Oracle High Availability Service started on node ludo.
2012-10-22 18:54:22.374
[ohasd(9097)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2012-10-22 18:54:24.722
[oracle/app/11.2.0/grid/bin/orarootagent.bin(9667)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/acfsload" spawned by agent "/oracle/app/11.2.0/grid/bin/orarootagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/ludo/agent/ohasd/orarootagent_root/orarootagent_root.log"
2012-10-22 18:54:37.443
[ohasd(9097)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-10-22 18:54:37.474
[gpnpd(9787)]CRS-2328:GPNPD started on node ludo.
2012-10-22 18:54:39.873
[cssd(9856)]CRS-1713:CSSD daemon is started in clustered mode
2012-10-22 18:54:41.664
[ohasd(9097)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2012-10-22 18:54:45.132
[cssd(9856)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /oracle/app/11.2.0/grid/log/ludo/cssd/ocssd.log
2012-10-22 18:55:00.151
[cssd(9856)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /oracle/app/11.2.0/grid/log/ludo/cssd/ocssd.log
2012-10-22 18:55:15.171
[cssd(9856)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /oracle/app/11.2.0/grid/log/ludo/cssd/ocssd.log
2012-10-22 18:55:30.190
[cssd(9856)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /oracle/app/11.2.0/grid/log/ludo/cssd/ocssd.log
2012-10-22 18:55:45.209
[cssd(9856)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /oracle/app/11.2.0/grid/log/ludo/cssd/ocssd.log
2012-10-22 18:56:00.228
[cssd(9856)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /oracle/app/11.2.0/grid/log/ludo/cssd/ocssd.log
2012-10-22 18:56:15.247
[cssd(9856)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /oracle/app/11.2.0/grid/log/ludo/cssd/ocssd.log


If I try to start up asm via sqlplus I get this error:

ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service


I don't understand why doesn't ASM get started since CSSD daemon is started.

Any help appreciated.

Thanks
  • 1. Re: unable to start crs
    Levi-Pereira Guru
    Currently Being Moderated
    What's the output of this log "/oracle/app/11.2.0/grid/log/ludo/cssd/ocssd.log" ?
  • 2. Re: unable to start crs
    585179 Expert
    Currently Being Moderated
    Hi,

    Do Check
    - If the disk permission and owner is still correct
    - Check if the disk is still visible not missing

    Cheers
  • 3. Re: unable to start crs
    AlleT Explorer
    Currently Being Moderated
    This is the output

    2012-10-17 17:58:30.500: [    CSSD][1124597024]clsu_load_ENV_levels: Module = CSSD, LogLevel = 2, TraceLevel = 0
    2012-10-17 17:58:30.500: [    CSSD][1124597024]clsu_load_ENV_levels: Module = GIPCNM, LogLevel = 2, TraceLevel = 0
    2012-10-17 17:58:30.500: [    CSSD][1124597024]clsu_load_ENV_levels: Module = GIPCGM, LogLevel = 2, TraceLevel = 0
    2012-10-17 17:58:30.500: [    CSSD][1124597024]clsu_load_ENV_levels: Module = GIPCCM, LogLevel = 2, TraceLevel = 0
    2012-10-17 17:58:30.500: [    CSSD][1124597024]clsu_load_ENV_levels: Module = CLSF, LogLevel = 0, TraceLevel = 0
    2012-10-17 17:58:30.500: [    CSSD][1124597024]clsu_load_ENV_levels: Module = SKGFD, LogLevel = 0, TraceLevel = 0
    2012-10-17 17:58:30.500: [    CSSD][1124597024]clsu_load_ENV_levels: Module = GPNP, LogLevel = 1, TraceLevel = 0
    2012-10-17 17:58:30.500: [    CSSD][1124597024]clsu_load_ENV_levels: Module = OLR, LogLevel = 0, TraceLevel = 0
    [    CSSD][1124597024]clsugetconf : Configuration type [4].
    2012-10-17 17:58:30.500: [    CSSD][1124597024]clssscmain: Starting CSS daemon, version 11.2.0.3.0, in (exclusive) mode with uniqueness value 1350489510
    2012-10-17 17:58:30.501: [    CSSD][1124597024]clssscmain: Environment is production
    2012-10-17 17:58:30.501: [    CSSD][1124597024]clssscmain: Core file size limit extended
    2012-10-17 17:58:30.507: [    CSSD][1124597024]clssscmain: GIPCHA down 0
    2012-10-17 17:58:30.508: [    CSSD][1124597024]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
    2012-10-17 17:58:30.508: [    CSSD][1124597024]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
    2012-10-17 17:58:30.508: [    CSSD][1124597024]clssscExtendLimits: The current soft limit for locked memory is 4294967295, hard limit is 4294967295
    2012-10-17 17:58:30.509: [    CSSD][1124597024]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
    2012-10-17 17:58:30.509: [    CSSD][1124597024]clssscSetPrivEnv: Setting priority to 4
    2012-10-17 17:58:31.522: [    CSSD][1124597024]clssscSetPrivEnv: IPMI IP address for this node is: 10.224.10.103
    [   CLWAL][1124597024]clsw_Initialize: OLR initlevel [70000]
    2012-10-17 17:58:31.679: [    CSSD][1124597024]clssscmain: Running as user oracle
    2012-10-17 17:58:31.681: [    CSSD][1124597024]clssscmain: RT queue setting is at default value
    2012-10-17 17:58:31.682: [    CSSD][1124597024]clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
    2012-10-17 17:58:31.682: [    CSSD][1124597024]clssgmInitCMInfoMin: clsmonJoined set via eXclusive mode
    [  clsdmt][1093880128]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=ludoDBG_CSSD))
    2012-10-17 17:58:31.682: [  clsdmt][1093880128]PID for the Process [16727], connkey 4
    2012-10-17 17:58:31.683: [    CSSD][1124597024]clssscGetParameterOLR: OLR fetch for parameter diagwait (14) failed with rc 21
    2012-10-17 17:58:31.687: [    CSSD][1124597024]clssnmInitNMInfoMin: Initializing first-reconfig to (0)
    2012-10-17 17:58:31.687: [    CSSD][1124597024]clssscmain: initgminfo done
    2012-10-17 17:58:31.688: [    CSSD][1090804032]clssgmclientlsnr: Spawned
    2012-10-17 17:58:31.688: [    CSSD][1090804032]clssgmEvtInformation: reqtype (13) cmProc ((nil)) client ((nil))
    2012-10-17 17:58:31.688: [    CSSD][1090804032]clssgmEvtInformation: reqtype (13) req (0x10c96bc0)
    2012-10-17 17:58:31.688: [    CSSD][1090804032]clssnmQueueNotification: type (13) 0x10c96bc0
    2012-10-17 17:58:31.689: [    CSSD][1090804032]clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ludo_)(GIPCID=00000000-00000000-16727))
    2012-10-17 17:58:31.689: [    GPNP][1124597024]clsgpnp_Init: [at clsgpnp0.c:585] '/oracle/app/11.2.0/grid' in effect as GPnP home base.
    2012-10-17 17:58:31.689: [    GPNP][1124597024]clsgpnp_Init: [at clsgpnp0.c:619] GPnP pid=16727, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3
    2012-10-17 17:58:31.700: [    GPNP][1124597024]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /oracle/app/11.2.0/grid/gpnp/ludo/wallets/peer/

    [   CLWAL][1124597024]clsw_Initialize: OLR initlevel [70000]
    2012-10-17 17:58:31.741: [    GPNP][1124597024]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url "ipc://GPNPD_ludo" disco "" [f=0 claimed- host: cname: seq: auth:]
    2012-10-17 17:58:31.753: [    GPNP][1124597024]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri "http://www.grid-pnp.org/2005/12/gpnp-errors#"]
    2012-10-17 17:58:32.317: [    CSSD][1090804032]clssscSelect: cookie accept request 0x2aaaac029fb0
    2012-10-17 17:58:32.317: [    CSSD][1090804032]clssgmAllocProc: (0x10cf5a00) allocated
    2012-10-17 17:58:32.318: [    CSSD][1090804032]clssgmClientConnectMsg: properties of cmProc 0x10cf5a00 - 1,2,3,4,5
    2012-10-17 17:58:32.318: [    CSSD][1090804032]clssgmClientConnectMsg: Connect from con(0x1b4) proc(0x10cf5a00) pid(16681) version 11:2:1:4, properties: 1,2,3,4,5
    2012-10-17 17:58:32.318: [    CSSD][1090804032]clssgmClientConnectMsg: The CSSD monitor is process (0x10cf5a00), number 1
    2012-10-17 17:58:32.318: [    CSSD][1090804032]clssgmEvtInformation: reqtype (11) cmProc (0x10cf5a00) client ((nil))
    2012-10-17 17:58:32.318: [    CSSD][1090804032]clssgmEvtInformation: reqtype (11) req (0x10cf9280)
    2012-10-17 17:58:32.318: [    CSSD][1090804032]clssnmQueueNotification: type (11) 0x10cf9280
    ...skipping...
    2012-10-22 20:47:26.389: [    CSSD][1099270464]clssgmDestroyProc: cleaning up proc(0xc5c71d0) con(0x9a33) skgpid ospid 9800 with 0 clients, refcount 0
    2012-10-22 20:47:26.389: [    CSSD][1099270464]clssgmDiscEndpcl: gipcDestroy 0x9a33
    2012-10-22 20:47:26.586: [    CSSD][1099270464]clssscSelect: cookie accept request 0xc38aab0
    2012-10-22 20:47:26.586: [    CSSD][1099270464]clssscevtypSHRCON: getting client with cmproc 0xc38aab0
    2012-10-22 20:47:26.586: [    CSSD][1099270464]clssgmRegisterClient: proc(3/0xc38aab0), client(595/0xc5ce680)
    2012-10-22 20:47:26.587: [    CSSD][1099270464]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0xc38aab0) client(0xc5ce680)
    2012-10-22 20:47:26.587: [    CSSD][1099270464]clssgmDiscEndpcl: gipcDestroy 0x9b17
    2012-10-22 20:47:26.594: [    CSSD][1099270464]clssscSelect: cookie accept request 0xc4fc580
    2012-10-22 20:47:26.594: [    CSSD][1099270464]clssscevtypSHRCON: getting client with cmproc 0xc4fc580
    2012-10-22 20:47:26.594: [    CSSD][1099270464]clssgmRegisterClient: proc(4/0xc4fc580), client(595/0xc38c440)
    2012-10-22 20:47:26.595: [    CSSD][1099270464]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0xc4fc580) client(0xc38c440)
    2012-10-22 20:47:26.595: [    CSSD][1099270464]clssgmDiscEndpcl: gipcDestroy 0x9b2d
    2012-10-22 20:47:26.738: [    CSSD][1099270464]clssscSelect: cookie accept request 0x2aaaac02a3e0
    2012-10-22 20:47:26.738: [    CSSD][1099270464]clssgmAllocProc: (0xc5bf5c0) allocated
    2012-10-22 20:47:26.738: [    CSSD][1099270464]clssgmClientConnectMsg: properties of cmProc 0xc5bf5c0 - 1,2,3,4,5
    2012-10-22 20:47:26.738: [    CSSD][1099270464]clssgmClientConnectMsg: Connect from con(0x9b70) proc(0xc5bf5c0) pid(9800) version 11:2:1:4, properties: 1,2,3,4,5
    2012-10-22 20:47:26.738: [    CSSD][1099270464]clssgmClientConnectMsg: msg flags 0x0000
    2012-10-22 20:47:26.804: [    CSSD][1099270464]clssgmExecuteClientRequest: MAINT recvd from proc 2 (0xc4d7010)
    2012-10-22 20:47:26.804: [    CSSD][1099270464]clssgmShutDown: Received abortive shutdown request from client.
    2012-10-22 20:47:26.804: [    CSSD][1099270464]###################################
    2012-10-22 20:47:26.804: [    CSSD][1099270464]clssscExit: CSSD aborting from thread GMClientListener
    2012-10-22 20:47:26.804: [    CSSD][1099270464]###################################
    2012-10-22 20:47:26.804: [    CSSD][1099270464](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
    2012-10-22 20:47:26.804: [    CSSD][1099270464]clssgmUpdateEventValue: CmInfo State val 0, changes 1
    2012-10-22 20:47:26.804: [ GIPCLIB][1099270464] gipclibMapSearch: gipcMapSearch() -> gipcMapGetNodeAddr() failed: ret:gipcretKeyNotFound (36), ht:0xc311670, idxPtr:0x2aaaaaaae8c0, key:0x4183d9a0, flags:0x0
    2012-10-22 20:47:26.804: [GIPCXCPT][1099270464] gipcObjectLookupF [gipcPostF : gipc.c : 2008]: search found no matching oid 0000000000000000, ret gipcretKeyNotFound (36), ret gipcretInvalidObject (3)
    2012-10-22 20:47:26.804: [GIPCXCPT][1099270464] gipcPostF [clssgmCMShutdown : clssgm.c : 3888]: EXCEPTION[ ret gipcretInvalidObject (3) ] failed to post obj 0000000000000000, flags 0x0
    2012-10-22 20:47:29.646: [    GPNP][1100847424]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url "ipc://GPNPD_ludo" disco "" [f=0 claimed- host: cname: seq: auth:]
    2012-10-22 20:47:29.660: [    GPNP][1100847424]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_ludo" disco ""
    2012-10-22 20:47:29.660: [    CSSD][1100847424]clssnmReadDiscoveryProfile: voting file discovery string()
    2012-10-22 20:47:29.660: [    CSSD][1100847424]clssnmvDDiscThread: using discovery string for initial discovery
    2012-10-22 20:47:29.660: [   SKGFD][1100847424]Discovery with str::

    2012-10-22 20:47:29.660: [   SKGFD][1100847424]UFS discovery with ::

    2012-10-22 20:47:29.660: [   SKGFD][1100847424]OSS discovery with ::

    2012-10-22 20:47:29.661: [   SKGFD][1100847424]Discovery with asmlib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: str ::

    2012-10-22 20:47:29.661: [   SKGFD][1100847424]Fetching asmlib disk :ORCL:DATADG01:

    2012-10-22 20:47:29.661: [   SKGFD][1100847424]ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op asm_open error Operation not permitted
    )
    2012-10-22 20:47:29.661: [    CSSD][1100847424]clssnmvDiskVerify: Successful discovery of 0 disks
    2012-10-22 20:47:29.661: [    CSSD][1100847424]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
    2012-10-22 20:47:29.661: [    CSSD][1100847424]clssnmvFindInitialConfigs: No voting files found
    2012-10-22 20:47:29.661: [    CSSD][1100847424](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
  • 4. Re: unable to start crs
    AlleT Explorer
    Currently Being Moderated
    Do you mean permission on the ASM disk?


    [root@ludo ~]# ls -l /dev/mapper/datadg.1p1
    brw-r----- 1 oracle oinstall 253, 1 Oct 22 18:53 /dev/mapper/datadg.1p1
  • 5. Re: unable to start crs
    585179 Expert
    Currently Being Moderated
    Hi,

    Edit your oracleasm config file under /etc/sysconfig/oracleasm to something like below
    ORACLEASM_SCANORDER="dm"
    ORACLEASM_SCANEXCLUDE="sd"
    And reboot the node again


    Cheers
  • 6. Re: unable to start crs
    AlleT Explorer
    Currently Being Moderated
    I have been able to sove this issue by manually changing parameters ORACLEASM_SCANORDER and ORACLEASM_SCANEXCLUDE in /etc/sysconfig/oracleasm-devoracleasm.

    Thanks everybody

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points