1 Reply Latest reply: Nov 5, 2012 7:39 AM by Sebastian Solbach -Dba Community-Oracle RSS

    Failed to start Oracle Clusterware stack on node2

    972371
      Hi,

      I find problem when install Grid Infrastructure, node 2 failed to start clusterware. Please help me to solve this.

      on node1
      ==============================================================
      root@may50 # /u01/app/11.2.0/grid/root.sh
      Running Oracle 11g root.sh script...

      The following environment variables are set as:
      ORACLE_OWNER= oracle
      ORACLE_HOME= /u01/app/11.2.0/grid

      Enter the full pathname of the local bin directory: [usr/local/bin]:
      The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
      Copying dbhome to /usr/local/bin ...
      The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
      Copying oraenv to /usr/local/bin ...
      The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
      Copying coraenv to /usr/local/bin ...

      Entries will be added to the /var/opt/oracle/oratab file as needed by
      Database Configuration Assistant when a database is created
      Finished running generic part of root.sh script.
      Now product-specific root actions will be performed.
      2000-01-01 09:57:09: Parsing the host name
      2000-01-01 09:57:09: Checking for super user privileges
      2000-01-01 09:57:09: User has super user privileges
      Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
      Creating trace directory
      LOCAL ADD MODE
      Creating OCR keys for user 'root', privgrp 'root'..
      Operation successful.
      root wallet
      root wallet cert
      root cert export
      peer wallet
      profile reader wallet
      pa wallet
      peer wallet keys
      pa wallet keys
      peer cert request
      pa cert request
      peer cert
      pa cert
      peer root cert TP
      profile reader root cert TP
      pa root cert TP
      peer pa cert TP
      pa peer cert TP
      profile reader pa cert TP
      profile reader peer cert TP
      peer user cert
      pa user cert
      Adding daemon to inittab
      Jan 1 09:58:54 may50 root: exec /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib /u01/app/11.2.0/grid/bin/crswrapexece.pl /u01/app/11.2.0/grid/crs/install/s_crsconfig_may50_env.txt /u01/app/11.2.0/grid/bin/ohasd.bin "reboot"
      CRS-4123: Oracle High Availability Services has been started.
      ohasd is starting
      Jan 1 09:59:27 may50 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [5180]: starting
      Jan 1 09:59:28 may50 mDNSResponder: Oracle mDNSResponder starting
      Jan 1 09:59:28 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 09:59:28 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 09:59:28 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 09:59:28 may50 mDNSResponder: Unable to parse DNS server list. Unicast DNS-SD unavailable
      CRS-2672: Attempting to start 'ora.gipcd' on 'may50'
      CRS-2672: Attempting to start 'ora.mdnsd' on 'may50'
      CRS-2676: Start of 'ora.gipcd' on 'may50' succeeded
      CRS-2676: Start of 'ora.mdnsd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.gpnpd' on 'may50'
      CRS-2676: Start of 'ora.gpnpd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.cssdmonitor' on 'may50'
      CRS-2676: Start of 'ora.cssdmonitor' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.cssd' on 'may50'
      CRS-2672: Attempting to start 'ora.diskmon' on 'may50'
      CRS-2676: Start of 'ora.diskmon' on 'may50' succeeded
      CRS-2676: Start of 'ora.cssd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.ctssd' on 'may50'
      CRS-2676: Start of 'ora.ctssd' on 'may50' succeeded

      ASM created and started successfully.

      DiskGroup DATA created successfully.

      clscfg: -install mode specified
      Successfully accumulated necessary OCR keys.
      Creating OCR keys for user 'root', privgrp 'root'..
      Operation successful.
      CRS-2672: Attempting to start 'ora.crsd' on 'may50'
      CRS-2676: Start of 'ora.crsd' on 'may50' succeeded
      CRS-4256: Updating the profile
      Successful addition of voting disk 103138bfd53b4f84bfb7c36adfeeb442.
      Successful addition of voting disk 8b7d610bedc74f65bf91b46ff4a816ec.
      Successful addition of voting disk 6d360ad323a64fd9bf66ff5638e6ede6.
      Successfully replaced voting disk group with +DATA.
      CRS-4256: Updating the profile
      CRS-4266: Voting file(s) successfully replaced
      ## STATE File Universal Id File Name Disk group
      -- ----- ----------------- --------- ---------
      1. ONLINE 103138bfd53b4f84bfb7c36adfeeb442 (/dev/rdsk/c0t40d2s3) [DATA]
      2. ONLINE 8b7d610bedc74f65bf91b46ff4a816ec (/dev/rdsk/c0t40d2s4) [DATA]
      3. ONLINE 6d360ad323a64fd9bf66ff5638e6ede6 (/dev/rdsk/c0t40d2s5) [DATA]
      Located 3 voting disk(s).
      CRS-2673: Attempting to stop 'ora.crsd' on 'may50'
      CRS-2677: Stop of 'ora.crsd' on 'may50' succeeded
      CRS-2673: Attempting to stop 'ora.asm' on 'may50'
      CRS-2677: Stop of 'ora.asm' on 'may50' succeeded
      CRS-2673: Attempting to stop 'ora.ctssd' on 'may50'
      CRS-2677: Stop of 'ora.ctssd' on 'may50' succeeded
      CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'may50'
      CRS-2677: Stop of 'ora.cssdmonitor' on 'may50' succeeded
      CRS-2673: Attempting to stop 'ora.cssd' on 'may50'
      CRS-2677: Stop of 'ora.cssd' on 'may50' succeeded
      CRS-2673: Attempting to stop 'ora.gpnpd' on 'may50'
      CRS-2677: Stop of 'ora.gpnpd' on 'may50' succeeded
      CRS-2673: Attempting to stop 'ora.gipcd' on 'may50'
      CRS-2677: Stop of 'ora.gipcd' on 'may50' succeeded
      CRS-2673: Attempting to stop 'ora.mdnsd' on 'may50'
      Jan 1 10:01:45 may50 mDNSResponder: clsdm requested mdnsd exit
      Jan 1 10:01:45 may50 mDNSResponder: mdnsd exit
      Jan 1 10:01:48 may50 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [5180]: stopping
      CRS-2677: Stop of 'ora.mdnsd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.mdnsd' on 'may50'
      Jan 1 10:01:51 may50 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [5679]: starting
      Jan 1 10:01:51 may50 mDNSResponder: Oracle mDNSResponder starting
      Jan 1 10:01:51 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:01:51 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:01:51 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:01:51 may50 mDNSResponder: Unable to parse DNS server list. Unicast DNS-SD unavailable
      CRS-2676: Start of 'ora.mdnsd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.gipcd' on 'may50'
      CRS-2676: Start of 'ora.gipcd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.gpnpd' on 'may50'
      CRS-2676: Start of 'ora.gpnpd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.cssdmonitor' on 'may50'
      CRS-2676: Start of 'ora.cssdmonitor' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.cssd' on 'may50'
      CRS-2672: Attempting to start 'ora.diskmon' on 'may50'
      CRS-2676: Start of 'ora.diskmon' on 'may50' succeeded
      CRS-2676: Start of 'ora.cssd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.ctssd' on 'may50'
      CRS-2676: Start of 'ora.ctssd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.asm' on 'may50'
      CRS-2676: Start of 'ora.asm' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.crsd' on 'may50'
      CRS-2676: Start of 'ora.crsd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.evmd' on 'may50'
      CRS-2676: Start of 'ora.evmd' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.asm' on 'may50'
      CRS-2676: Start of 'ora.asm' on 'may50' succeeded
      CRS-2672: Attempting to start 'ora.DATA.dg' on 'may50'
      CRS-2676: Start of 'ora.DATA.dg' on 'may50' succeeded
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
      Jan 1 10:04:47 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:2 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:2 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:1 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce0:2 AF=2
      Jan 1 10:05:07 may50 mDNSResponder: mdnsd found interface name=ce2 AF=2

      may50 2000/01/01 10:05:15 /u01/app/11.2.0/grid/cdata/may50/backup_20000101_100515.olr
      Configure Oracle Grid Infrastructure for a Cluster ... succeeded
      Updating inventory properties for clusterware
      Starting Oracle Universal Installer...

      Checking swap space: must be greater than 500 MB. Actual 14511 MB Passed
      The inventory pointer is located at /var/opt/oracle/oraInst.loc
      The inventory is located at /u01/app/oraInventory
      'UpdateNodeList' was successful.
      root@may50 #
      ==============================================================

      on node2
      ==============================================================
      bash-3.2# /u01/app/11.2.0/grid/root.sh
      Running Oracle 11g root.sh script...

      The following environment variables are set as:
      ORACLE_OWNER= oracle
      ORACLE_HOME= /u01/app/11.2.0/grid

      Enter the full pathname of the local bin directory: [usr/local/bin]:
      The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
      Copying dbhome to /usr/local/bin ...
      The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
      Copying oraenv to /usr/local/bin ...
      The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y
      Copying coraenv to /usr/local/bin ...

      Entries will be added to the /var/opt/oracle/oratab file as needed by
      Database Configuration Assistant when a database is created
      Finished running generic part of root.sh script.
      Now product-specific root actions will be performed.
      2000-01-01 10:01:50: Parsing the host name
      2000-01-01 10:01:50: Checking for super user privileges
      2000-01-01 10:01:50: User has super user privileges
      Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
      Creating trace directory
      LOCAL ADD MODE
      Creating OCR keys for user 'root', privgrp 'root'..
      Operation successful.
      Adding daemon to inittab
      Jan 1 10:02:00 may52 root: exec /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib /u01/app/11.2.0/grid/bin/crswrapexece.pl /u01/app/11.2.0/grid/crs/install/s_crsconfig_may52_env.txt /u01/app/11.2.0/grid/bin/ohasd.bin "reboot"
      CRS-4123: Oracle High Availability Services has been started.
      ohasd is starting
      Jan 1 10:02:34 may52 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [3613]: starting
      Jan 1 10:02:34 may52 mDNSResponder: Oracle mDNSResponder starting
      Jan 1 10:02:34 may52 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:02:34 may52 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:02:34 may52 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:02:34 may52 mDNSResponder: Unable to parse DNS server list. Unicast DNS-SD unavailable
      CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node may50, number 1, and is terminating
      CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'may52'
      CRS-2677: Stop of 'ora.cssdmonitor' on 'may52' succeeded
      CRS-2673: Attempting to stop 'ora.gpnpd' on 'may52'
      CRS-2677: Stop of 'ora.gpnpd' on 'may52' succeeded
      CRS-2673: Attempting to stop 'ora.gipcd' on 'may52'
      CRS-2677: Stop of 'ora.gipcd' on 'may52' succeeded
      CRS-2673: Attempting to stop 'ora.mdnsd' on 'may52'
      Jan 1 10:03:34 may52 mDNSResponder: clsdm requested mdnsd exit
      Jan 1 10:03:34 may52 mDNSResponder: mdnsd exit
      Jan 1 10:03:37 may52 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [3613]: stopping
      CRS-2677: Stop of 'ora.mdnsd' on 'may52' succeeded
      An active cluster was found during exclusive startup, restarting to join the cluster
      CRS-2672: Attempting to start 'ora.mdnsd' on 'may52'
      Jan 1 10:03:45 may52 mDNSResponder (Engineering Build) (Nov 2 2009 05:02:07) [3775]: starting
      Jan 1 10:03:46 may52 mDNSResponder: Oracle mDNSResponder starting
      Jan 1 10:03:46 may52 mDNSResponder: mdnsd found interface name=lo0 AF=2
      Jan 1 10:03:46 may52 mDNSResponder: mdnsd found interface name=ce0 AF=2
      Jan 1 10:03:46 may52 mDNSResponder: mdnsd found interface name=ce2 AF=2
      Jan 1 10:03:46 may52 mDNSResponder: Unable to parse DNS server list. Unicast DNS-SD unavailable
      CRS-2676: Start of 'ora.mdnsd' on 'may52' succeeded
      CRS-2672: Attempting to start 'ora.gipcd' on 'may52'
      CRS-2676: Start of 'ora.gipcd' on 'may52' succeeded
      CRS-2672: Attempting to start 'ora.gpnpd' on 'may52'
      CRS-2676: Start of 'ora.gpnpd' on 'may52' succeeded
      CRS-2672: Attempting to start 'ora.cssdmonitor' on 'may52'
      CRS-2676: Start of 'ora.cssdmonitor' on 'may52' succeeded
      CRS-2672: Attempting to start 'ora.cssd' on 'may52'
      CRS-2672: Attempting to start 'ora.diskmon' on 'may52'
      CRS-2674: Start of 'ora.diskmon' on 'may52' failed
      CRS-2679: Attempting to clean 'ora.diskmon' on 'may52'
      CRS-2674: Start of 'ora.cssd' on 'may52' failed
      CRS-2679: Attempting to clean 'ora.cssd' on 'may52'
      CRS-2681: Clean of 'ora.cssd' on 'may52' succeeded
      CRS-2679: Attempting to clean 'ora.diskmon' on 'may52'
      CRS-2680: Clean of 'ora.diskmon' on 'may52' failed
      CRS-2681: Clean of 'ora.diskmon' on 'may52' succeeded
      CRS-4000: Command Start failed, or completed with errors.
      CRS-2672: Attempting to start 'ora.cssd' on 'may52'
      CRS-2672: Attempting to start 'ora.diskmon' on 'may52'
      CRS-2674: Start of 'ora.diskmon' on 'may52' failed
      CRS-2679: Attempting to clean 'ora.diskmon' on 'may52'
      CRS-5016: Process "/u01/app/11.2.0/grid/bin/diskmon" spawned by agent "/u01/app/11.2.0/grid/bin/orarootagent.bin" for action "clean" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/may52/agent/ohasd/orarootagent_root/orarootagent_root.log"
      CRS-2681: Clean of 'ora.diskmon' on 'may52' succeeded
      CRS-2674: Start of 'ora.cssd' on 'may52' failed
      CRS-2679: Attempting to clean 'ora.cssd' on 'may52'
      CRS-2678: 'ora.cssd' on 'may52' has experienced an unrecoverable failure
      CRS-0267: Human intervention required to resume its availability.
      CRS-4000: Command Start failed, or completed with errors.
      Command return code of 1 (256) from command: /u01/app/11.2.0/grid/bin/crsctl start resource ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE
      Start of resource "ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE" failed
      Failed to start CTSS
      Failed to start Oracle Clusterware stack
      bash-3.2#
      ==============================================================

      Thank you.

      Tuan
        • 1. Re: Failed to start Oracle Clusterware stack on node2
          Sebastian Solbach -Dba Community-Oracle
          Hi,

          there seems something wrong with your network setup.

          Especially this:
          mDNSResponder: Unable to parse DNS server list. Unicast DNS-SD unavailable
          indicates a problem with resolving network addresses etc.

          So please check your network setup/hostfiles and resolv.conf
          Also make sure you don't use special characters in the hostname.

          More information should also be found in ocssd.log

          Maybe if you look for the error there, you can find something in MyOracleSupport.

          Regards
          Sebastian