1 2 Previous Next 23 Replies Latest reply: Jul 28, 2011 3:26 PM by DD RSS

    EM not working after restart

    DD
      I have an inherited installation of OEM that I cannot get to work. By which I mean I get a "503 Service Unavailable Service is not initialized correctly. Verify that the repository connection information provided is correct" error in my browser. This might have been caused when I tried to change the sysman password. I think OEM was still running after the pw change, but after a database restart (application of Oracle patch) it has stayed broken.

      I have only been working with Oracle for a few months, I do not depend on OEM for a lot but I need to get OEM working again. I have googled for answers and read documentation, but I am missing something and I am hoping one of you all will know what it is.

      OEM version: 11.2.0.1.0 (agent) Oracle Enterprise Manager 10g Release 5 Grid Control (oms)
      database version: 11.2.0.1.0

      I have seem to have three oracle homes:
      db & listener: /u01/db/oracle/product/11.2.0/dbhome_1
      agent: /u01/app/oracle/agent10g
      oms: /u01/app/oracle/oms10g

      OS: linux

      I have attempted several times to drop and create the repository but end up with the same errors. I don't think the drops and creates have been successful though, because of errors during the process and becuase the sysman user keeps getting locked.

      The repository errors look like these:
      May 24, 2011 4:23:14 PM oracle.sysman.emcp.EMConfig perform
      
      INFO: This operation is being logged at /u01/db/oracle/cfgtoollogs/emca/sid01/emca_2011_05_24_16_22_01.log.
      
      May 24, 2011 4:23:15 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
      
      INFO: Stopping Database Control (this may take a while) ...
      
      May 24, 2011 4:23:36 PM oracle.sysman.emcp.EMReposConfig createRepository
      
      INFO: Creating the EM repository (this may take a while) ...
      
      May 24, 2011 4:23:36 PM oracle.sysman.emcp.EMReposConfig invoke
      
      SEVERE: Error creating the repository
      
      May 24, 2011 4:23:36 PM oracle.sysman.emcp.EMReposConfig invoke
      
      INFO: Refer to the log file at /u01/db/oracle/cfgtoollogs/emca/sid01/emca_repos_create_<date>.log for more details.
      
      May 24, 2011 4:23:36 PM oracle.sysman.emcp.EMConfig perform
      
      SEVERE: Error creating the repository
      
      Refer to the log file at /u01/db/oracle/cfgtoollogs/emca/sid01/emca_2011_05_24_16_22_01.log for more details.
      
      Could not complete the configuration. Refer to the log file at /u01/db/oracle/cfgtoollogs/emca/sid01/emca_2011_05_24_16_22_01.log for more details.
      Edited by: user13716631 on May 25, 2011 2:14 PM
        • 1. Re: EM not working after restart
          EricvdS
          I think you are trying to install DBConsole (the 1 database smaller version of Grid Control) and, since you have a OMS home, not EM Grid Control.
          Check the last part of http://askdba.org/weblog/2009/07/installing-10-2-0-5-em-using-software-only-on-10-2-0-411g-db-gui/
          Step 7 since you already have the OMS software installed. I don't know if it will work but it's worth a try.

          Also take a look at the Oracle documentaion on the Softwaare only and Configure later installation type: http://otndnld.oracle.co.jp/document/products/V17375-01/doc/install.102/e10953/installing_em.htm#CCHGBDCB

          First stop the agent and check if any crontab job is scheduled that connects to the user sysman, if so disable that job. That could be the reason sysman gets locked.

          Eric
          • 2. Re: EM not working after restart
            DD
            Hi Eric, Thanks for the reply.

            First, I checked crontab and commented out the only cronjob that might have affected this. I also stopped dbconsole. The sysman user continues to get locked repeatedly. A good idea to check this anyway, I'm sure.

            About your other suggestions, should I do only one of these, or both:

            1. to do a Software Only Installation
            2. reconfigure the existing installation like in: http://askdba.org/weblog/2009/07/installing-10-2-0-5-em-using-software-only-on-10-2-0-411g-db-gui/

            Are these actions to resolve a software conflict? I ask this because I have not tried to install anything on this server so far and this software used to work.

            David
            • 3. Re: EM not working after restart
              Vishal Joshi
              Hi,

              Have you checked your log files?. Kindly go to oms home and then sysman and log. Kindly check and paste the log files here if it has any error.

              Thanks,
              Vishal Joshi
              Oracle Apps DBA.
              • 4. Re: EM not working after restart
                DD
                Hi Vishal,

                Here are the most recent logs with errors:

                emoms.log - these errors repeat over and over
                2011-05-26 08:46:10,126 [ApplicationServerThread-16] ERROR app.SessionObjectManager sessionDestroyed.128 - java.sql.SQLException: ORA-28000: the account is locked
                
                java.sql.SQLException: ORA-28000: the account is locked
                
                        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:137)
                        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:315)
                        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:276)
                        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:271)
                        at oracle.jdbc.driver.T4CTTIoauthenticate.receiveOauth(T4CTTIoauthenticate.java:646)
                        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:307)
                        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:429)
                        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:150)
                        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:31)
                        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:608)
                        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:217)
                        at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:113)
                        at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:76)
                        at oracle.jdbc.pool.OracleImplicitConnectionCache.makeCacheConnection(OracleImplicitConnectionCache.java:1361)
                        at oracle.jdbc.pool.OracleImplicitConnectionCache.getCacheConnection(OracleImplicitConnectionCache.java:440)
                        at oracle.jdbc.pool.OracleImplicitConnectionCache.getConnection(OracleImplicitConnectionCache.java:335)
                        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:285)
                        at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:253)
                        at oracle.sysman.util.jdbc.ConnectionCache._getConnection(ConnectionCache.java:336)
                        at oracle.sysman.util.jdbc.ConnectionCache._getConnection(ConnectionCache.java:322)
                        at oracle.sysman.util.jdbc.ConnectionCache.getUnwrappedConnection(ConnectionCache.java:575)
                        at oracle.sysman.emSDK.svc.conn.FGAConnectionCache.getFGAConnection(FGAConnectionCache.java:218)
                        at oracle.sysman.emSDK.svc.conn.ConnectionService.getPrivateConnection(ConnectionService.java:1158)
                        at oracle.sysman.emSDK.svc.conn.ConnectionService.getPrivateConnection(ConnectionService.java:1192)
                        at oracle.sysman.eml.app.SessionObjectManager.sessionDestroyed(SessionObjectManager.java:116)
                        at oracle.sysman.eml.app.SessionListener.sessionDestroyed(SessionListener.java:114)
                        at com.evermind.server.http.HttpApplication.invalidateSession(HttpApplication.java:683)
                        at com.evermind.server.http.EvermindHttpSession.invalidate(EvermindHttpSession.java:272)
                        at com.evermind.server.http.SessionAsyncInvalidateHandler.invalidateSession(SessionAsyncInvalidateHandler.java:173)
                        at com.evermind.server.http.SessionAsyncInvalidateHandler.run(SessionAsyncInvalidateHandler.java:133)
                        at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
                        at java.lang.Thread.run(Thread.java:534)
                
                2011-05-26 08:46:13,759 [AJPRequestHandler-ApplicationServerThread-11] ERROR conn.ConnectionService verifyRepositoryEx.911 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
                
                2011-05-26 08:46:23,837 [AJPRequestHandler-ApplicationServerThread-5] ERROR conn.ConnectionService verifyRepositoryEx.911 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
                
                2011-05-26 08:46:25,068 [AJPRequestHandler-ApplicationServerThread-10] ERROR conn.ConnectionService verifyRepositoryEx.911 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
                
                2011-05-26 08:46:27,676 [AJPRequestHandler-ApplicationServerThread-15] ERROR conn.ConnectionService verifyRepositoryEx.911 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
                
                2011-05-26 08:46:33,368 [AJPRequestHandler-ApplicationServerThread-10] ERROR conn.ConnectionService verifyRepositoryEx.911 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
                
                2011-05-26 08:46:35,561 [AJPRequestHandler-ApplicationServerThread-7] ERROR conn.ConnectionService verifyRepositoryEx.911 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
                
                2011-05-26 08:46:36,588 [AJPRequestHandler-ApplicationServerThread-6] ERROR conn.ConnectionService verifyRepositoryEx.911 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
                
                2011-05-26 08:46:37,933 [AJPRequestHandler-ApplicationServerThread-12] ERROR conn.ConnectionService verifyRepositoryEx.911 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
                
                2011-05-26 08:46:41,041 [ApplicationServerThread-16] ERROR app.SessionObjectManager sessionDestroyed.128 - java.sql.SQLException: ORA-28000: the account is locked
                emdctl.trc
                2011-05-02 11:03:31 Thread-4138851088 ERROR http: 6: snmehl_read: error on read: Connection reset by peer (error = 104)
                2011-05-02 11:03:32 Thread-4138253072 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-02 11:03:32 Thread-4138457872 WARN  http: snmehl_connect: connect failed to (server01.something.com:1156): Connection refused (error = 111)
                2011-05-09 13:14:44 Thread-4138048272 WARN  http: snmehl_connect: connect failed to (server01.something.com:1156): Connection refused (error = 111)
                2011-05-09 13:15:03 Thread-4138470160 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-09 13:20:24 Thread-4138756880 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-09 13:23:46 Thread-4138195728 WARN  http: snmehl_connect: connect failed to (server01.something.com:1156): Connection refused (error = 111)
                2011-05-10 16:39:19 Thread-4138027792 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-10 16:43:15 Thread-4138162960 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-10 16:54:23 Thread-4138691344 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-25 13:19:12 Thread-4138584848 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-25 15:06:51 Thread-4138253072 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-25 15:10:25 Thread-4138466064 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-25 15:11:15 Thread-4138810128 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                2011-05-25 15:13:37 Thread-4138064656 WARN  http: snmehl_connect: connect failed to (server01.something.com:1157): Connection refused (error = 111)
                Thanks for your response
                • 5. Re: EM not working after restart
                  mnazim-Oracle
                  It seems your sysman/dbsnmp account is unlock, please check
                  • 6. Re: EM not working after restart
                    DD
                    Hi mnazim ,

                    Yes, the sysman account is locked. Every time I unlock it, it gets locked again with a minute or so - no matter how many times I unlock it. I do not know how to stop it from locking.

                    Thanks
                    • 7. Re: EM not working after restart
                      mnazim-Oracle
                      Please follow the below steps and chelc

                      emctl stop dbconsole

                      alter user sysman identified by <password> account unlock;

                      Edit the file $ORACLE_HOME/host_sid/sysman/config/emoms.properties


                      Locate and change following lines:
                      oracle.sysman.eml.mntr.emdRepPwd=<encrypted_string>
                      oracle.sysman.eml.mntr.emdRepPwdEncrypted=TRUE

                      to

                      oracle.sysman.eml.mntr.emdRepPwd=<password_from_repository_unecrypted>
                      oracle.sysman.eml.mntr.emdRepPwdEncrypted=FALSE


                      emctl start dbconsole
                      • 8. Re: EM not working after restart
                        DD
                        Hi, followed your suggestion but sysman still got locked within a couple of minutes. I have tried this before with the same results. Could this be because of multiple emoms.properties files or is this OK?
                        find . -name emoms.properties
                        ./u01/app/oracle/oms10g/inventory/Templates/sysman/config/emoms.properties
                        ./u01/app/oracle/oms10g/sysman/config/emoms.properties
                        ./u01/db/oracle/product/11.2.0/dbhome_1/server01.something.com_sid01/sysman/config/emoms.properties
                        ./u01/db/oracle/product/11.2.0/dbhome_1/inventory/Templates/sysman/config/emoms.properties
                        ./u01/db/oracle/product/11.2.0/dbhome_1/sysman/config/emoms.properties
                        ./u01/db/oracle/product/10.2.0/db_1/server01.something.com_sid02/sysman/config/emoms.properties
                        I edited /u01/db/oracle/product/11.2.0/dbhome_1/server01.something.com_sid01/sysman/config/emoms.properties.

                        Thanks

                        Edited by: user13716631 on May 26, 2011 11:39 AM

                        Edited by: user13716631 on May 26, 2011 11:40 AM
                        • 9. Re: EM not working after restart
                          DD
                          I got several errors after changing the password and restarting dbconsole (in addition to the ones above):

                          from /u01/db/oracle/product/11.2.0/dbhome_1/server01.something.edu_sid01/sysman/log/emagent_perl.trc
                          several of these:
                          alertlogAdr.pl: Thu May 26 14:40:47 2011: WARN:  First stack timestamp for /u01/db/oracle/diag/rdbms/sid01/sid01/alert/log.xml: 2011-05-07T22:42:28.523-04:00
                          and these;
                          maxRowCount.pl: Thu May 26 14:31:14 2011: ERROR:  Max Count Value not set properly in file /u01/db/oracle/product/11.2.0/dbhome_1/sysman/config/esa/database.properties line no. 77
                          from /u01/db/oracle/product/11.2.0/dbhome_1/server01.something.edu_sid01/sysman/log/emdb.nohup
                          several of these:
                          WARNING: Code-source /u01/db/oracle/product/11.2.0/dbhome_1/lib/xml.jar (from <classpath> in /u01/db/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/oc4j_applications/applications/em/em) has the same filename but is not identical to /u01/db/oracle/product/11.2.0/dbhome_1/oc4j/lib/xml.jar (from <code-source> (ignore manifest Class-Path) in META-INF/boot.xml in /u01/db/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/home/oc4j.jar). If it contains different versions of the same classes, it will be masked as the latter is already visible in the search path of loader em.web.em:0.0.0.
                          from /u01/db/oracle/product/11.2.0/dbhome_1/server01.something.edu_sid01/sysman/log/emagentfetchlet.log
                          and many of these:
                          2011-05-26 14:31:23,426 [main] WARN  track.OracleInventory mineOneOffPatches.1910 - ECM: File name "/u01/app/oracle/oms10g/sysman/jlib/emDB.jar : /oracle/sysman/emo/dist/streams/apply/ApplyAdminObject.class" is the same as a file name already registered in home /u01/app/oracle/oms10g.  This file is ignored.
                          2011-05-26 14:31:23,426 [main] WARN  track.OracleInventory mineOneOffPatches.1910 - ECM: File name "/u01/app/oracle/oms10g/sysman/jlib/emDB.jar : /oracle/sysman/emo/perf/bean/autosta/AutoSqlTuneBean.class" is the same as a file name already registered in home /u01/app/oracle/oms10g.  This file is ignored.
                          2011-05-26 14:31:23,434 [main] WARN  track.OracleInventory mineOneOffPatches.1910 - ECM: File name "/u01/app/oracle/oms10g/sysman/admin/emdrep/sql/core/latest/collections/collections_util_pkgbody.sql" is the same as a file name already registered in home /u01/app/oracle/oms10g.  This file is ignored.
                          from /u01/db/oracle/product/11.2.0/dbhome_1/server01.something.edu_sid01/sysman/log/emagent.trc
                          011-05-26 14:31:15,300 Thread-1127131456 ERROR : (nmecmgr.c,3230):Memory 0x0 encountered, expect struct_id=11011
                          2011-05-26 14:31:15,407 Thread-1127131456 ERROR : (nmecmgr.c,3230):Memory 0x0 encountered, expect struct_id=11011
                          2011-05-26 14:31:15,538 Thread-184593792 WARN  collector: the column name first_message_age in this condition does not exist in metric aq_msgs_persistentq_per_subscriber
                          2011-05-26 14:31:15,541 Thread-184593792 WARN  collector: the column name first_message_age in this condition does not exist in metric aq_msgs_persistentq_per_subscriber
                          2011-05-26 14:31:15,597 Thread-184593792 WARN  collector: the column name first_message_age in this condition does not exist in metric aq_msgs_persistentq_per_subscriber
                          2011-05-26 14:31:15,608 Thread-1130281280 WARN  http: nmehl_connect_internal: connect failed to (server01.something.edu:1158): Connection refused (error = 111)
                          2011-05-26 14:31:15,608 Thread-1130281280 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://server01.something.edu:1158/em/upload/: retStatus=-1
                          2011-05-26 14:31:15,609 Thread-1130281280 WARN  http: nmehl_connect_internal: connect failed to (server01.something.edu:1158): Connection refused (error = 111)
                          2011-05-26 14:31:15,609 Thread-1130281280 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://server01.something.edu:1158/em/upload/: retStatus=-1
                          2011-05-26 14:31:16,059 Thread-1114532160 WARN  recvlets.aq: no matching registered alert type for instance_efficiency.pxdwngrd_ps
                          2011-05-26 14:31:16,060 Thread-1114532160 WARN  recvlets.aq: no matching registered alert type for instance_efficiency.pxdwngrdserial_pt
                          2011-05-26 14:31:16,062 Thread-1114532160 WARN  recvlets.aq: no matching registered alert type for wait_bottlenecks.userio_wait_cnt
                          2011-05-26 14:31:16,062 Thread-1114532160 WARN  recvlets.aq: no matching registered alert type for wait_bottlenecks.other_wait_cnt
                          2011-05-26 14:31:16,424 Thread-1084442944 WARN  http: nmehl_connect_internal: connect failed to (server01.something.edu:1158): Connection refused (error = 111)
                          2011-05-26 14:31:16,424 Thread-1084442944 ERROR command: nmejcn: failed http connection to http://server01.something.edu:1158/em/upload/: retStatus=-1
                          Edited by: user13716631 on May 26, 2011 11:59 AM
                          • 10. Re: EM not working after restart
                            EricvdS
                            In my first post I asked if you are using DBConsole or EM Grid Control. You do have a OMS home so I think you want to run EM Grid Control. You are now trying to get DBConsole running.
                            • 11. Re: EM not working after restart
                              DD
                              I thought DBconsole was part of EM. But if I you understand you correctly, I should not be trying to get DBconsole working, but get Grid Control running instead. I didn't even know that Grid Control and EM were different things.

                              So, I should do one of these:

                              1. to do a Software Only Installation
                              2. reconfigure the existing installation like in: http://askdba.org/weblog/2009/07/installing-10-2-0-5-em-using-software-only-on-10-2-0-411g-db-gui/

                              ?

                              Thanks
                              • 12. Re: EM not working after restart
                                DD
                                I ran the ConfigureGC.pl script.
                                Changed the sysman password and unlocked and - it works! for the first time in a long time.
                                opmnctl startall
                                start agent
                                go to EM url, new error:
                                503 Service Unavailable
                                Service is not initialized correctly. OMS version 10.2.0.5.0 is not compatible with repository version 10.2.0.4.2.
                                So I need to upgrade the repository? If so, do I use the software only installation?

                                Edited by: user13716631 on May 27, 2011 10:47 AM
                                • 13. Re: EM not working after restart
                                  EricvdS
                                  During running the ConfigureGC.pl something must have gone wrong. You can check on errors in
                                  /u01/app/oms10g/cfgtoollogs/oui/configActions2011<date and time>.log

                                  If you see errors that indicate that objects already exist see http://yannickjaquier.wordpress.com/2010/03/26/grid-control-10-2-0-5-installation-on-rhel-5-4-x64/ the part about How To Drop, Create And Recreate DB Control In A 10g Database. You have to start with a "clean" database, meaning without sysman and sysman related objects.

                                  What I said about DBConsole and EM Grid Control is actually Database Control and Grid Control. If you want to monitor just the (one) database you can use Database Control. If you have more databases (and other targets like os, application server, Weblogic etc etc etc) to monitor then you go for Grid Control. I think you first need to know what you want to use.

                                  The software is already installed so you only have to configure it.

                                  Eric
                                  • 14. Re: EM not working after restart
                                    DD
                                    Hi Eric, log contents below.

                                    configActions2011-05-27_10-29-51-AM.log:
                                    ###################################################
                                    The action configuration is performing
                                    The action configuration is performing
                                    ###################################################
                                    configActions2011-05-27_10-30-00-AM.log:
                                    ###################################################
                                    The action patchsetConfiguration is performing
                                    The action patchsetConfiguration has successfully completed
                                    ###################################################
                                    .err logs are empty.

                                    Should I drop and create repository anyway?

                                    Thanks for the clarification on grid vs. database control. I will be monitoring several other Oracle Db servers (and this is what it was used for before it went down), so I want Grid Control.

                                    David
                                    1 2 Previous Next