5 Replies Latest reply: Jul 4, 2012 10:31 AM by Osama_Mustafa RSS

    Grid on a standalone server Cluster daemons not starting during server boot

    joe_irungu
      Hi!I have installed Oracle Grid on a standalone server and setup Oracle db 11.2.0.2 on Oracle Linux 6.2 64 bit server.When I reoot the server and run crs_stat -t,several daemons havent started thus the ASM and db instances are also down as below

      Name Type Target State Host
      ------------------------------------------------------------
      ora.DATA.dg ora....up.type OFFLINE OFFLINE
      ora.FRADG.dg ora....up.type OFFLINE OFFLINE
      ora....ER.lsnr ora....er.type ONLINE ONLINE amldb01dc
      ora.amldb.db ora....se.type OFFLINE OFFLINE
      ora.asm ora.asm.type OFFLINE OFFLINE
      ora.cssd ora.cssd.type ONLINE OFFLINE
      ora.diskmon ora....on.type ONLINE OFFLINE
      ora.evmd ora.evm.type ONLINE ONLINE amldb01dc
      ora.ons ora.ons.type OFFLINE OFFLINE

      I am forced to manually start the daemons via command crsctl start resource -all then I manually start the ASM and db instances.
      Yet when I run the commands

      crsctl config has
      CRS-4622: Oracle High Availability Services autostart is enabled.

      crsctl check has
      CRS-4638: Oracle High Availability Services is online.

      Thus I would assume the daemons would start automatically during boot.

      How can I resolve this?
        • 1. Re: Grid on a standalone server Cluster daemons not starting during server boot
          Osama_Mustafa
          hi

          Post $GRID_HOME/log/SID/alert*.log and crsd.log
          and you can give this try

          crsctl stop cluster
          crsctl start cluster
          crsctl check crs

          crs_stat -t

          post output Please


          BR
          Osama
          • 2. Re: Grid on a standalone server Cluster daemons not starting during server boot
            415289
            When I reoot the server and run crs_stat -t,several daemons havent started thus the ASM and db instances are also down as below
            wait for few mins,they will comeup automatically.
            check crs status after reboot by crsctl check crs.
            fyi,-crs_stat is deprecated in 11gR2. Re: Replacement for crs_stat -p ?
            • 3. Re: Grid on a standalone server Cluster daemons not starting during server boot
              joe_irungu
              crstcl check crs seems not to be applicable to single instance but for RAC,since I have a single instance db,the output of the command is as below

              crsctl check crs
              Parse error:
              'crs' is an invalid argument

              Brief usage:
              crsctl check has
              Check status of OHAS.

              There is no crsd.log in path /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/crsd.The output of crs_stat -t is already posted in initial post and output of entire alert log is as below

              2012-07-03 16:18:35.975
              [client(8182)]CRS-2101:The OLR was formatted using version 3.
              2012-07-03 16:18:36.471
              [client(8218)]CRS-1001:The OCR was formatted using version 3.
              [client(8293)]CRS-10001:CRS-6021: No msg for has:crs-6021 [l][unlimited]
              [client(8294)]CRS-10001:CRS-6021: No msg for has:crs-6021 [n][65536]
              2012-07-03 16:18:42.528
              [ohasd(8291)]CRS-2112:The OLR service started on node amldb01dc.
              2012-07-03 16:18:42.538
              [ohasd(8291)]CRS-1301:Oracle High Availability Service started on node amldb01dc.
              2012-07-03 16:18:51.271
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(8465)]CRS-5815:Agent '/u01/app/oracle/product/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:1:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-03 16:18:52.299
              [evmd(8479)]CRS-1401:EVMD started on node amldb01dc.
              [client(8520)]CRS-10001:03-Jul-12 16:18 ACFS-9459: ADVM/ACFS is not supported on this OS version: 'error: file /etc/SuSE-release: No such file or directory
              '
              [client(8889)]CRS-10001:03-Jul-12 16:19 ACFS-9459: ADVM/ACFS is not supported on this OS version: 'error: file /etc/SuSE-release: No such file or directory
              '
              2012-07-03 16:19:22.600
              [cssd(9014)]CRS-1713:CSSD daemon is started in local-only mode
              2012-07-03 16:19:31.128
              [cssd(9014)]CRS-1601:CSSD Reconfiguration complete. Active nodes are amldb01dc .
              [client(10038)]CRS-10001:03-Jul-12 16:25 ACFS-9459: ADVM/ACFS is not supported on this OS version: 'error: file /etc/SuSE-release: No such file or directory
              '
              2012-07-03 16:34:13.791
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(8465)]CRS-5010:Update of configuration file "/u01/app/oracle/product/11.2.0/dbhome_1/srvm/admin/oratab.bak.amldb01dc" failed: details at "(:CLSN00011:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-03 16:37:39.674
              [ohasd(8291)]CRS-2765:Resource 'ora.DATA.dg' has failed on server 'amldb01dc'.
              2012-07-03 16:37:43.636
              [ohasd(8291)]CRS-2767:Resource state recovery not attempted for 'ora.asm' as its target state is OFFLINE
              2012-07-03 16:38:06.905
              [cssd(9014)]CRS-1603:CSSD on node amldb01dc shutdown by user.
              2012-07-03 16:38:07.006
              [cssd(9014)]CRS-1660:The CSS daemon shutdown has completed
              2012-07-03 16:38:10.118
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(8465)]CRS-5016:Process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-03 16:42:09.578
              [ohasd(4827)]CRS-2112:The OLR service started on node amldb01dc.
              2012-07-03 16:42:09.649
              [ohasd(4827)]CRS-1301:Oracle High Availability Service started on node amldb01dc.
              2012-07-03 16:42:10.437
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4992)]CRS-5815:Agent '/u01/app/oracle/product/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:2:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-03 16:42:10.667
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4992)]CRS-5016:Process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-03 16:42:40.463
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4992)]CRS-5818:Aborted command 'check for resource: ora.DATA.dg amldb01dc 1' for resource 'ora.DATA.dg'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-03 16:42:40.464
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4992)]CRS-5818:Aborted command 'check for resource: ora.FRADG.dg amldb01dc 1' for resource 'ora.FRADG.dg'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-03 16:42:54.641
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(5046)]CRS-5815:Agent '/u01/app/oracle/product/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:4:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-03 16:42:55.675
              [evmd(5060)]CRS-1401:EVMD started on node amldb01dc.
              2012-07-03 16:54:35.840
              [cssd(5256)]CRS-1713:CSSD daemon is started in local-only mode
              2012-07-03 16:54:44.396
              [cssd(5256)]CRS-1601:CSSD Reconfiguration complete. Active nodes are amldb01dc .
              2012-07-03 16:54:56.832
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(5046)]CRS-5010:Update of configuration file "/u01/app/oracle/product/11.2.0/dbhome_1/srvm/admin/oratab.bak.amldb01dc" failed: details at "(:CLSN00011:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-04 10:03:53.530
              [ohasd(4827)]CRS-2767:Resource state recovery not attempted for 'ora.asm' as its target state is OFFLINE
              2012-07-04 10:04:02.701
              [cssd(5256)]CRS-1603:CSSD on node amldb01dc shutdown by user.
              2012-07-04 10:04:02.809
              [cssd(5256)]CRS-1660:The CSS daemon shutdown has completed
              2012-07-04 10:04:03.008
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(5046)]CRS-5016:Process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-04 10:07:48.492
              [ohasd(4698)]CRS-2112:The OLR service started on node amldb01dc.
              2012-07-04 10:07:48.534
              [ohasd(4698)]CRS-1301:Oracle High Availability Service started on node amldb01dc.
              2012-07-04 10:07:49.059
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4946)]CRS-5815:Agent '/u01/app/oracle/product/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:1:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:07:49.192
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4946)]CRS-5016:Process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-04 10:08:19.085
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4946)]CRS-5818:Aborted command 'check for resource: ora.DATA.dg amldb01dc 1' for resource 'ora.DATA.dg'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:08:19.093
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4946)]CRS-5818:Aborted command 'check for resource: ora.FRADG.dg amldb01dc 1' for resource 'ora.FRADG.dg'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:08:33.278
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(5053)]CRS-5815:Agent '/u01/app/oracle/product/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:4:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:08:34.319
              [evmd(5069)]CRS-1401:EVMD started on node amldb01dc.
              2012-07-04 10:27:13.716
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(5053)]CRS-5016:Process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-04 10:30:57.260
              [ohasd(4619)]CRS-2112:The OLR service started on node amldb01dc.
              2012-07-04 10:30:57.280
              [ohasd(4619)]CRS-1301:Oracle High Availability Service started on node amldb01dc.
              2012-07-04 10:30:57.660
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4992)]CRS-5815:Agent '/u01/app/oracle/product/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:1:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:30:57.784
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4992)]CRS-5016:Process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-04 10:31:27.685
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4992)]CRS-5818:Aborted command 'check for resource: ora.FRADG.dg amldb01dc 1' for resource 'ora.FRADG.dg'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:31:27.685
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4992)]CRS-5818:Aborted command 'check for resource: ora.DATA.dg amldb01dc 1' for resource 'ora.DATA.dg'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:31:41.868
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(5168)]CRS-5815:Agent '/u01/app/oracle/product/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:4:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:31:42.916
              [evmd(5184)]CRS-1401:EVMD started on node amldb01dc.
              2012-07-04 10:39:17.166
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(5168)]CRS-5016:Process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-04 10:43:01.219
              [ohasd(4575)]CRS-2112:The OLR service started on node amldb01dc.
              2012-07-04 10:43:01.240
              [ohasd(4575)]CRS-1301:Oracle High Availability Service started on node amldb01dc.
              2012-07-04 10:43:01.643
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4971)]CRS-5815:Agent '/u01/app/oracle/product/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:1:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:43:01.781
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4971)]CRS-5016:Process "/u01/app/oracle/product/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log"
              2012-07-04 10:43:31.680
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4971)]CRS-5818:Aborted command 'check for resource: ora.FRADG.dg amldb01dc 1' for resource 'ora.FRADG.dg'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:43:31.680
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(4971)]CRS-5818:Aborted command 'check for resource: ora.DATA.dg amldb01dc 1' for resource 'ora.DATA.dg'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:43:45.864
              [u01/app/oracle/product/11.2.0/grid/bin/oraagent.bin(5120)]CRS-5815:Agent '/u01/app/oracle/product/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:4:2} in /u01/app/oracle/product/11.2.0/grid/log/amldb01dc/agent/ohasd/oraagent_grid/oraagent_grid.log.
              2012-07-04 10:43:46.915
              [evmd(5136)]CRS-1401:EVMD started on node amldb01dc.
              • 4. Re: Grid on a standalone server Cluster daemons not starting during server boot
                joe_irungu
                Daemons did not come up after even 2 hours.also crstcl check crs seems not to be applicable to single instance but for RAC,since I have a single instance db,the output of the command is as below

                crsctl check crs
                Parse error:
                'crs' is an invalid argument

                Brief usage:
                crsctl check has
                Check status of OHAS.
                • 5. Re: Grid on a standalone server Cluster daemons not starting during server boot
                  Osama_Mustafa
                  Try

                  crsctl disable crs
                  crsctl disable has


                  crsctl enable has
                  crsctl enable crs

                  Wait for 2 min

                  crsctl check crs

                  crs_stat -t

                  Br
                  Osama