4 Replies Latest reply: Jul 27, 2010 4:11 PM by Satishbabu Gunukula RSS

    emca -deconfig dbcontrol db -repos drop, perl 100%CPU

    dmitry_rpd
      Hello!

      My system is: Red Hat Enterprise Linux AS release 4 x64, Oracle 10.2.0.1.0

      Few days ago I had found that 2 perl processes eat 100% CPU.
      So I decided to restart EM. But the problem remain the same.

      -----
      Then I decided to recreate EM repository:

      emca -deconfig dbcontrol db -repos drop -SID db1 -HOST HP -ORACLE_HOME /app/oracle

      STARTED EMCA at Jul 9, 2010 11:49:17 AM
      EM Configuration Assistant, Version 10.2.0.1.0 Production
      Copyright (c) 2003, 2005, Oracle. All rights reserved.

      Enter the following information:
      Listener port number: 1521
      Password for SYS user:
      Password for SYSMAN user:

      Do you wish to continue? [yes(Y)/no(N)]: y
      Jul 9, 2010 11:49:31 AM oracle.sysman.emcp.EMConfig perform
      INFO: This operation is being logged at /app/oracle/cfgtoollogs/emca/ db1 /emca_2010-07-09_11-49-17-AM.log.
      Jul 9, 2010 11:49:31 AM oracle.sysman.emcp.util.DBControlUtil stopOMS
      INFO: Stopping Database Control (this may take a while) ...
      Jul 9, 2010 11:49:33 AM oracle.sysman.emcp.EMReposConfig dropRepository
      INFO: Dropping the EM repository (this may take a while) ...

      In this point command hangs.
      -----
      Looking to the processes:
      ps aux | grep perl

      oracle 15592 101 0.0 23752 8672 pts/2 R+ 12:11 0:24 /app/oracle/perl/bin/perl /app/oracle/sysman/admin/emdrep/bin/emrepmgr.pl -connect (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=HP)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME= db1))) -repos_user SYSMAN -action drop -verbose -output_file /app/oracle/cfgtoollogs/emca/ db1 /emca_repos_drop_2010-07-09_12-11-40-PM.log
      oracle 15667 0.0 0.0 51100 688 pts/4 S+ 12:12 0:00 grep perl
      -----
      Looking to the log:
      [oracle@HP db1]$ tail -n 50 /app/oracle/cfgtoollogs/emca/ db1 /emca_repos_drop_2010-07-09_12-11-40-PM.log
      [09-07-2010 12:11:40] Enter SYS user's password :
      [09-07-2010 12:11:40]
      [09-07-2010 12:11:40] Enter repository user password :
      [09-07-2010 12:11:40]
      [09-07-2010 12:11:40] Getting temporary tablespace from database...
      -----
      Could you please help me with this?
      Why does perl hang? And how to correctly recreate repository?
      (Is it possible to recreate repository without any outages(quiescing or restart the DB))
      Thanks!
        • 1. Re: emca -deconfig dbcontrol db -repos drop, perl 100%CPU
          dmitry_rpd
          Yesterday I've started EM, but emagent had 100%CUP again.
          And there was an active session by sysman:
          BEGIN EMD_NOTIFICATION.QUEUE_READY(:1, :2, :3); END;

          After making some research, i've done this:
          sqlplus sysman@db1
          SQL>@/app/oracle/sysman/admin/emdrep/sql/core/latest/notification/notification_pkgbodys.sql

          (changing string DBMS_AQ.LISTEN (agents, qtimeout_in, agent);
          to DBMS_AQ.LISTEN (agents, 1200, agent);
          in it
          )
          After restarting EM this problem remain the same.
          Logs:
          cat ./emagent.log
          2010-07-12 12:28:44 Thread-4136454368 Starting Agent 10.1.0.4.1 from /app/oracle (00701)
          2010-07-12 12:29:44 Thread-4136454368 target {db1, oracle_database} is broken: cannot compute dynamic properties in time. (00155)
          2010-07-12 12:29:44 Thread-4136454368 EMAgent started successfully (00702)

          and

          [oracle@HP log]$ tail -n 5 ./emdctl.trc
          2010-07-12 12:28:32 Thread-4136453824 WARN http: snmehl_connect: connect failed to (HP:3938): Connection refused (error = 111)
          2010-07-12 12:28:34 Thread-4136453824 WARN http: snmehl_connect: connect failed to (HP:3938): Connection refused (error = 111)
          2010-07-12 12:28:37 Thread-4136453824 WARN http: snmehl_connect: connect failed to (HP:3938): Connection refused (error = 111)
          2010-07-12 12:28:40 Thread-4136453824 WARN http: snmehl_connect: connect failed to (HP:3938): Connection refused (error = 111)
          2010-07-12 12:28:43 Thread-4136453824 WARN http: snmehl_connect: connect failed to (HP:3938): Connection refused (error = 111)

          Why does "Connection refused" appear?

          How to deal with this issue?
          Yours suggestions will be VERY appreciated. Thanks!

          Edited by: dmitry_rpd on Jul 13, 2010 12:33 AM
          • 2. Re: emca -deconfig dbcontrol db -repos drop, perl 100%CPU
            Satishbabu Gunukula
            Couple of months back i have faced same issue and i have opened a TAR, but no help from Oracle.

            But finally i figured out the issue, this problem occurs with 10.2.0.1

            If you didnt apply DST and your EM/GC running during DST period you will see that perl scripts using 100% CPU. This is because EM config files not able to adjust DST time change with system time change.

            Solution : restarting the server will resolve issue. (when you restart server EM/GC will read the system time and sync with that and you will not see any issues)

            Hope this helps,

            Regards
            http://www.oracleracexpert.com
            RAC database Instance hang/restart due to node eviction and Solution.
            http://www.oracleracexpert.com/2009/09/ora-29740-evicted-by-member-0-group.html
            IPC Send timeout issue and Solution.
            http://www.oracleracexpert.com/2009/09/ora-29740-evicted-by-member-0-group.html
            • 3. Re: emca -deconfig dbcontrol db -repos drop, perl 100%CPU
              dmitry_rpd
              Sorry for my latency with reply.
              And thank you very much for help!

              But... Is there any solution to adjust time settings without restarting the server (and stop the DB)?

              Thanks!
              • 4. Re: emca -deconfig dbcontrol db -repos drop, perl 100%CPU
                Satishbabu Gunukula
                I have tried all available options (NTP change, rebuilding EM rep, editing config files..etc) with out restarting the server ,but no luck.

                Finally figured out that when you stop DB services and restart the servers, all configuration files able to sync with system time.

                Hope this helps,

                Regards
                http://www.oracleracexpert.com
                Ttransportable-tablespace export and import on Same Endian platforms
                http://www.oracleracexpert.com/2009/08/transportable-tablespace-export-import.html]
                Remove Grid control agents or targets from repository
                http://www.oracleracexpert.com/2010/06/remove-grid-control-agents-or-targets.html