6 Replies Latest reply: Oct 22, 2012 7:50 AM by Red8 RSS

    Oracle RAC 11.2.0.3 installation on OEL 6.3 HAIP error

    Red8
      Hi friends

      i have these big problem, i tried to install RAC on these two virtual machines before i do the production enviroment and here are the problems i am facing
      =========================
      Operating system OEL 6.3
      GRID 11.2.0.3
      DB 11.2.0.3

      ===========================
      1. during pre requset test /etc/resolv.vonf failed saying PRVF-5636 : The DNS response time for an unreachable node can not be found . . .. but i cheked everything

      [root@ubsrac1 /]# nslookup ubsrac1
      Server: 192.168.0.100
      Address: 192.168.0.100#53

      Name: ubsrac1.kifiya.com
      Address: 192.168.0.117

      [root@ubsrac1 /]# nslookup ubsrac1-vip
      Server: 192.168.0.100
      Address: 192.168.0.100#53

      Name: ubsrac1-vip.kifiya.com
      Address: 192.168.0.119

      [root@ubsrac1 /]# nslookup ubsrac2
      Server: 192.168.0.100
      Address: 192.168.0.100#53

      Name: ubsrac2.kifiya.com
      Address: 192.168.0.118

      [root@ubsrac1 /]# nslookup ubsrac2-vip
      Server: 192.168.0.100
      Address: 192.168.0.100#53

      Name: ubsrac2-vip.kifiya.com
      Address: 192.168.0.120

      [root@ubsrac1 /]# nslookup ubsrac-scan
      Server: 192.168.0.100
      Address: 192.168.0.100#53

      Name: ubsrac-scan.kifiya.com
      Address: 192.168.0.139
      Name: ubsrac-scan.kifiya.com
      Address: 192.168.0.136
      Name: ubsrac-scan.kifiya.com
      Address: 192.168.0.138

      ====================================
      And for unknow host here is the output

      [root@ubsrac1 /]# time nslookup unknown-not-reachable-node
      Server: 192.168.0.100
      Address: 192.168.0.100#53

      ** server can't find unknown-not-reachable-node: SERVFAIL


      real 0m0.019s
      user 0m0.003s
      sys 0m0.008s
      [root@ubsrac1 /]#

      =====================================

      Everything is ok on both the nodes........... but still i get the PRVF-5636 : The DNS response time

      ==========================
      output of /etc/hosts
      [root@ubsrac1 /]# cat /etc/hosts
      127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
      ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

      #Network configuration for ubsrac1
      192.168.0.117 ubsrac1 ubsrac1.kifiya.com
      192.168.0.119 ubsrac1-vip ubsrac1-vip.kifiya.com
      172.16.1.17 ubsrac1-priv ubsrac1-priv.kifiya.com

      #Network configuration for ubsrac2
      192.168.0.118 ubsrac2 ubsrac2.kifiya.com
      192.168.0.120 ubsrac2-vip ubsrac2-vip.kifiya.com
      172.16.1.18 ubsrac2-priv ubsrac2-priv.kifiya.com

      =============================================

      and i ignored and went with installation and excuted root.sh on the first node and here is the last output where i get the HAIP error

      Adding Clusterware entries to upstart
      CRS-2672: Attempting to start 'ora.mdnsd' on 'ubsrac1'
      CRS-2676: Start of 'ora.mdnsd' on 'ubsrac1' succeeded
      CRS-2672: Attempting to start 'ora.gpnpd' on 'ubsrac1'
      CRS-2676: Start of 'ora.gpnpd' on 'ubsrac1' succeeded
      CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ubsrac1'
      CRS-2672: Attempting to start 'ora.gipcd' on 'ubsrac1'
      CRS-2676: Start of 'ora.cssdmonitor' on 'ubsrac1' succeeded
      CRS-2676: Start of 'ora.gipcd' on 'ubsrac1' succeeded
      CRS-2672: Attempting to start 'ora.cssd' on 'ubsrac1'
      CRS-2672: Attempting to start 'ora.diskmon' on 'ubsrac1'
      CRS-2676: Start of 'ora.diskmon' on 'ubsrac1' succeeded
      CRS-2676: Start of 'ora.cssd' on 'ubsrac1' succeeded

      ASM created and started successfully.

      Disk Group DATA created successfully.

      clscfg: -install mode specified
      Successfully accumulated necessary OCR keys.
      Creating OCR keys for user 'root', privgrp 'root'..
      Operation successful.
      CRS-4256: Updating the profile
      Successful addition of voting disk a745c85386eb4f4fbfa0f3b25c48a1e9.
      Successful addition of voting disk 146d859894f64f0cbf8172ecabd0e40a.
      Successful addition of voting disk 7c41353b913b4f6dbf1b5c56f9661e28.
      Successful addition of voting disk 5364b3e4d2f14fe9bf69207f38af44c5.
      Successful addition of voting disk 6376684a593a4fd1bf50e89459820d8c.
      Successfully replaced voting disk group with +DATA.
      CRS-4256: Updating the profile
      CRS-4266: Voting file(s) successfully replaced
      ## STATE File Universal Id File Name Disk group
      -- ----- ----------------- --------- ---------
      1. ONLINE a745c85386eb4f4fbfa0f3b25c48a1e9 (ORCL:CRSVOL1) [DATA]
      2. ONLINE 146d859894f64f0cbf8172ecabd0e40a (ORCL:CRSVOL2) [DATA]
      3. ONLINE 7c41353b913b4f6dbf1b5c56f9661e28 (ORCL:CRSVOL3) [DATA]
      4. ONLINE 5364b3e4d2f14fe9bf69207f38af44c5 (ORCL:CRSVOL4) [DATA]
      5. ONLINE 6376684a593a4fd1bf50e89459820d8c (ORCL:CRSVOL5) [DATA]
      Located 5 voting disk(s).
      Start of resource "ora.cluster_interconnect.haip" failed
      CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'ubsrac1'
      CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
      Start action for HAIP aborted. For details refer to "(:CLSN00107:)" in "/oragi/11.2.0.3/grid/log/ubsrac1/agent/ohasd/orarootagent_root/orarootagent_root.log".
      CRS-2674: Start of 'ora.cluster_interconnect.haip' on 'ubsrac1' failed
      CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'ubsrac1'
      CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'ubsrac1' succeeded
      CRS-4000: Command Start failed, or completed with errors.
      HAIP startup failure considered fatal, terminating at /oragi/11.2.0.3/grid/crs/install/crsconfig_lib.pm line 1263.
      /oragi/11.2.0.3/grid/perl/bin/perl -I/oragi/11.2.0.3/grid/perl/lib -I/oragi/11.2.0.3/grid/crs/install /oragi/11.2.0.3/grid/crs/install/rootcrs.pl execution failed


      ======================================================
      final out out is as follows

      [grid@ubsrac1 bin]$ ./crsctl stat res -t -init
      --------------------------------------------------------------------------------
      NAME TARGET STATE SERVER STATE_DETAILS
      --------------------------------------------------------------------------------
      Cluster Resources
      --------------------------------------------------------------------------------
      ora.asm
      1 OFFLINE OFFLINE Instance Shutdown
      ora.cluster_interconnect.haip
      1 ONLINE OFFLINE
      ora.crf
      1 ONLINE ONLINE ubsrac1
      ora.crsd
      1 OFFLINE OFFLINE EXCLUSIVE
      ora.cssd
      1 ONLINE ONLINE ubsrac1
      ora.cssdmonitor
      1 ONLINE ONLINE ubsrac1
      ora.ctssd
      1 ONLINE ONLINE ubsrac1 ACTIVE:0
      ora.diskmon
      1 OFFLINE OFFLINE
      ora.evmd
      1 OFFLINE OFFLINE
      ora.gipcd
      1 ONLINE ONLINE ubsrac1
      ora.gpnpd
      1 ONLINE ONLINE ubsrac1
      ora.mdnsd
      1 ONLINE ONLINE ubsrac1
      [grid@ubsrac1 bin]$


      please i need help . . .. i am stuck here i searched the web and tried things for days and i seem to run out of ideas and solutions

      any help will be greaatley aperciated ............


      Thanks in advance

      Regards
      Rediet
        • 1. Re: Oracle RAC 11.2.0.3 installation on OEL 6.3 HAIP error
          damorgan
          And thus another example of why Oracle does not certify and fully support RAC almost all VMs.

          Your listing clearly states
          Start action for HAIP aborted. For details refer to "(:CLSN00107:)" in "/oragi/11.2.0.3/grid/log/ubsrac1/agent/ohasd/orarootagent_root/orarootagent_root.log".
          How can we help you without the details?

          But please note also this appears to be an HAIP issue not an Oracle RAC issue so my recommendation would be that you contact IBM.
          • 2. Re: Oracle RAC 11.2.0.3 installation on OEL 6.3 HAIP error
            Levi Pereira
            Are you using HAIP? If yes you should check this tech note on MOS:

            *11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip [ID 1210883.1]*

            There is a list of bug that you should be aware.
            • 3. Re: Oracle RAC 11.2.0.3 installation on OEL 6.3 HAIP error
              user12028775
              Hi Dan,
              HAIP is a feature of Oracle Clusterware. You may have confused letter soup... IBM have HACMP.
              The acronyms seem as virus are multiplying with over the years.
              • 4. Re: Oracle RAC 11.2.0.3 installation on OEL 6.3 HAIP error
                damorgan
                You are correct ... my apology ... I had just finished helping someone with an AIX problem and AIX was on my mind.
                • 5. Re: Oracle RAC 11.2.0.3 installation on OEL 6.3 HAIP error
                  Red8
                  hi
                  I have read this document from MOS

                  11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip [ID 1210883.1]

                  still no luck


                  Please help ............ any ideas ....................

                  Regards

                  Edited by: Red8 on Oct 22, 2012 3:48 AM

                  Edited by: Red8 on Oct 22, 2012 3:53 AM
                  • 6. Re: Oracle RAC 11.2.0.3 installation on OEL 6.3 HAIP error
                    Red8
                    hi damorgan

                    here is the out put sir, . ................

                    ================================================
                    tail -200 /oragi/11.2.0.3/grid/log/ubsrac1/agent/ohasd/orarootagent_root/orarootagent_root.log

                    2012-10-22 08:22:33.539: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:22:33.539: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:23:00.074: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23943
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5520
                    2012-10-22 08:23:02.528: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5520], buflen=8
                    2012-10-22 08:23:02.528: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5520]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:23:03.542: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:23:03.542: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5520
                    2012-10-22 08:23:32.526: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5520], buflen=8
                    2012-10-22 08:23:32.526: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5520]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:23:33.550: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:23:33.550: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:24:00.077: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23947
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5520
                    2012-10-22 08:24:02.530: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5520], buflen=8
                    2012-10-22 08:24:02.530: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5520]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:24:02.531: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 3. Current # is 2
                    2012-10-22 08:24:03.554: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:24:03.554: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:24:30.088: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23951
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:24:32.542: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:24:32.542: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:24:33.553: [    AGFW][3097618208] Recvd request to shed the threads
                    2012-10-22 08:24:33.553: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 2. Current # is 3
                    2012-10-22 08:24:33.553: [CLSFRAME][3074782976] {0:11:5628} Worker thread is exiting in TM [MultiThread] to meet the desired count of 2. New count is 2
                    2012-10-22 08:24:33.557: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:24:33.557: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:25:02.551: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:25:02.551: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:25:03.565: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:25:03.565: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:25:30.091: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23955
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:25:32.554: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:25:32.554: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:25:33.567: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:25:33.567: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:26:00.103: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23959
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:26:02.556: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:26:02.556: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:26:03.569: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:26:03.569: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:26:32.562: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:26:32.562: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:26:32.563: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 3. Current # is 2
                    2012-10-22 08:26:33.576: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:26:33.576: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:27:00.106: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23963
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5440
                    2012-10-22 08:27:02.569: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5440], buflen=8
                    2012-10-22 08:27:02.570: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5440]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:27:03.581: [    AGFW][3097618208] Recvd request to shed the threads
                    2012-10-22 08:27:03.581: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 2. Current # is 3
                    2012-10-22 08:27:03.581: [CLSFRAME][3076884224] {0:11:5630} Worker thread is exiting in TM [MultiThread] to meet the desired count of 2. New count is 2
                    2012-10-22 08:27:03.583: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:27:03.583: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:27:30.117: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23967
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5440
                    2012-10-22 08:27:32.570: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5440], buflen=8
                    2012-10-22 08:27:32.570: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5440]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:27:33.584: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:27:33.585: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:27:33.585: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 3. Current # is 2
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5440
                    2012-10-22 08:28:02.580: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5440], buflen=8
                    2012-10-22 08:28:02.580: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5440]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:28:03.593: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:28:03.595: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:28:30.120: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23971
                    2012-10-22 08:28:32.581: [    AGFW][3097618208] Recvd request to shed the threads
                    2012-10-22 08:28:32.581: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 2. Current # is 3
                    2012-10-22 08:28:32.581: [CLSFRAME][3074782976] {0:11:5632} Worker thread is exiting in TM [MultiThread] to meet the desired count of 2. New count is 2
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:28:32.584: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:28:32.584: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:28:33.598: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:28:33.598: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:28:33.600: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 3. Current # is 2
                    2012-10-22 08:29:00.132: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23975
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:29:02.585: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:29:02.585: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:29:03.600: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:29:03.600: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:29:30.145: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23977
                    2012-10-22 08:29:32.586: [    AGFW][3097618208] Recvd request to shed the threads
                    2012-10-22 08:29:32.586: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 2. Current # is 3
                    2012-10-22 08:29:32.586: [CLSFRAME][3076884224] {0:11:5634} Worker thread is exiting in TM [MultiThread] to meet the desired count of 2. New count is 2
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5520
                    2012-10-22 08:29:32.589: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5520], buflen=8
                    2012-10-22 08:29:32.589: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5520]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:29:32.590: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 3. Current # is 2
                    2012-10-22 08:29:33.603: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:29:33.603: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:30:02.596: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:30:02.597: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:30:03.608: [    AGFW][3097618208] Recvd request to shed the threads
                    2012-10-22 08:30:03.608: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 2. Current # is 3
                    2012-10-22 08:30:03.608: [CLSFRAME][3074782976] {0:11:5636} Worker thread is exiting in TM [MultiThread] to meet the desired count of 2. New count is 2
                    2012-10-22 08:30:03.612: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:30:03.612: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:30:30.147: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23983
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:30:32.601: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:30:32.601: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:30:33.615: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:30:33.616: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:31:00.158: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23987
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:31:02.601: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:31:02.601: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:31:03.616: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:31:03.616: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:31:32.610: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:31:32.610: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:31:33.625: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:31:33.625: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:32:00.160: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23991
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:32:02.613: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:32:02.613: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:32:03.628: [ora.crf][3076884224] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:32:03.628: [ora.crf][3076884224] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:32:03.628: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 3. Current # is 2
                    2012-10-22 08:32:30.173: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23995
                    [  clsdmc][3076884224]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=A4000930
                    2012-10-22 08:32:32.616: [ora.ctssd][3076884224] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9ba4000930], buflen=8
                    2012-10-22 08:32:32.616: [ora.ctssd][3076884224] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9ba4000930]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:32:33.630: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:32:33.630: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:33:02.622: [    AGFW][3097618208] Recvd request to shed the threads
                    2012-10-22 08:33:02.622: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 2. Current # is 3
                    2012-10-22 08:33:02.622: [CLSFRAME][3076884224] {0:11:5638} Worker thread is exiting in TM [MultiThread] to meet the desired count of 2. New count is 2
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5520
                    2012-10-22 08:33:02.625: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5520], buflen=8
                    2012-10-22 08:33:02.625: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5520]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:33:03.639: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:33:03.639: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:33:30.177: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:23999
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5520
                    2012-10-22 08:33:32.631: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5520], buflen=8
                    2012-10-22 08:33:32.631: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5520]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:33:33.635: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:33:33.635: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:34:00.177: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:24003
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5520
                    2012-10-22 08:34:02.631: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5520], buflen=8
                    2012-10-22 08:34:02.631: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5520]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:34:03.646: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:34:03.646: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:34:30.184: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:24007
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5520
                    2012-10-22 08:34:32.637: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5520], buflen=8
                    2012-10-22 08:34:32.637: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5520]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:34:33.651: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:34:33.651: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    2012-10-22 08:35:00.188: [    AGFW][3072681728] {0:0:185} Agent received the message: AGENT_HB[Engine] ID 12293:24009
                    [  clsdmc][3074782976]CLSDMC.C returnbuflen=8, extraDataBuf=CC, returnbuf=B00D5520
                    2012-10-22 08:35:02.641: [ora.ctssd][3074782976] {0:0:185} [check] clsdmc_respget return: status=0, ecode=0, returnbuf=[0x7f9bb00d5520], buflen=8
                    2012-10-22 08:35:02.642: [ora.ctssd][3074782976] {0:0:185} [check] translateReturnCodes, return = 0, state detail = ACTIVE:0Checkcb data [0x7f9bb00d5520]: mode[0xcc] offset[0 ms].
                    2012-10-22 08:35:02.642: [CLSFRAME][3097618208] TM [MultiThread] is changing desired thread # to 3. Current # is 2
                    2012-10-22 08:35:03.656: [ora.crf][3074782976] {0:0:193} [check] clsdmc_respget return: status=0, ecode=0
                    2012-10-22 08:35:03.656: [ora.crf][3074782976] {0:0:193} [check] Check return = 0, state detail = NULL
                    [root@ubsrac1 ~]#
                    ================================

                    please take a look at it


                    Thanks in advance

                    Edited by: Red8 on Oct 22, 2012 5:50 AM