9 Replies Latest reply: Nov 4, 2013 11:11 AM by EdSpn RSS

    OMS doesn't start - can't connect to database

    EdSpn

      Hi all,

       

      My dev/test environment won't start OMS anymore. This really has us stopped in our tracks...

      It will start the WebTier, but OMS won't come up:

       

      [oracle@ora-emcc log]$ $OMS_HOME/bin/emctl status oms

      Oracle Enterprise Manager Cloud Control 12c Release 2

      Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.

      WebTier is Up

      Oracle Management Server is not functioning because of the following reason:

      Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.

       

      Trying to check oms details results in an error as well:

      [oracle@ora-emcc config]$ $OMS_HOME/bin/emctl status oms -details -sysman_pwd $SYSMAN_PWD

      Oracle Enterprise Manager Cloud Control 12c Release 2

      Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.

      Exception in thread "main" java.util.MissingResourceException: Can't find bundle for base name oracle.security.jps.internal.common.resources.keystore.KeyStoreResources, locale

              at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1427)

              at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1250)

              at java.util.ResourceBundle.getBundle(ResourceBundle.java:952)

              at oracle.security.jps.util.JpsBundle.<init>(JpsBundle.java:70)

              at oracle.security.jps.util.JpsBundle.getResourceBundle(JpsBundle.java:151)

              at oracle.security.jps.util.JpsBundle.getResourceBundle(JpsBundle.java:106)

              at oracle.security.jps.JpsLocalizableException.getResourceBundle(JpsLocalizableException.java:157)

              at oracle.security.jps.JpsLocalizableException.getLocalizedMessage(JpsLocalizableException.java:140)

              at oracle.security.jps.JpsLocalizableException.getLocalizedMessage(JpsLocalizableException.java:131)

              at java.lang.Throwable.toString(Throwable.java:343)

              at java.lang.String.valueOf(String.java:2826)

              at java.lang.StringBuilder.append(StringBuilder.java:115)

              at java.lang.Throwable.printStackTraceAsCause(Throwable.java:536)

              at java.lang.Throwable.printStackTraceAsCause(Throwable.java:545)

              at java.lang.Throwable.printStackTraceAsCause(Throwable.java:545)

              at java.lang.Throwable.printStackTrace(Throwable.java:516)

              at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:59)

              at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:342)

              at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:304)

              at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:234)

              at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)

              at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)

              at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)

              at org.apache.log4j.Category.callAppenders(Category.java:203)

              at org.apache.log4j.Category.forcedLog(Category.java:388)

              at org.apache.log4j.Category.error(Category.java:319)

              at oracle.sysman.emctl.oms.PrintOMSSetupInfo.main(PrintOMSSetupInfo.java:240)

       

      In the WebLogic Admin Console, I can see that BIP, EMGC_ADMINSERVER and EMGC_OMS1 are all in state Running and health Ok.

       

      Our OMS and OMR servers are (and have always been) 2 different machines. The database on OMR has always been (and is currently) up and available.

      Checking that I can connect to the repository database on OMR from the OMS server:

       

      [oracle@ora-emcc ~]$ sqlplus sysman/$SYSMAN_PWD@ora-emccdb:1521/emccdb

       

       

      SQL*Plus: Release 11.1.0.7.0 - Production on Thu Oct 10 23:34:53 2013

       

       

      Copyright (c) 1982, 2008, Oracle.  All rights reserved.

       

       

       

       

      Connected to:

      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

      With the Partitioning, OLAP, Data Mining and Real Application Testing options

       

       

      SQL>

       

       

      Any suggestions will be much appreciated.

       

      Ed

        • 1. Re: OMS doesn't start - can't connect to database
          User729207-Oracle

          Pls take a a look at emoms.trc, emoms.log and EMGC_OMS1.out

          One of these should have more details.

          • 2. Re: OMS doesn't start - can't connect to database
            Rahul-Em-Oracle

            The below two files should give the exact root cause for the issue

            One of the known issue is if the Repository database archive log file system is full , and ORA-00257: archiver error is recorded in the below files

            ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out

            ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_ADMINSERVER.out

            Also Repository DB alert.log file should also be reviewed

             

            If Repos DB is clean then we can check the ../gc_inst/em/EMGC_OMS1/sysman/log/* for any security/wallet related issue

             

            You can also open a SR with EM team if the above does not help

            • 3. Re: OMS doesn't start - can't connect to database
              Bip

              The 'emkey' may not be working currently in your environment.

               

              Run the command "emctl status emkey" from location <12c_OMS_HOME>/bin/

               

              If there is a issue with 'emkey' then you should be seeing an error as below,

               

              "The EMKey is not configured properly or is corrupted in the credential store and does not exist in the Management Repository"

               

              A quick solution in the above scenario would be to restore the 'emkey.ora' file from your file system backup if you have. The 'emkey' location in 12c OMS is <12c_OMS_HOME>/sysman/config/.

               

              Once you restore the file from your back up you will need to configure the emkey by running the below command,

               

              <12c_OMS_HOME>/bin/emctl config emkey -copy_to_credstore_from_file

               

              Let me know if this helps.

               

               

              Regards,

              Biplab

              • 4. Re: OMS doesn't start - can't connect to database
                Ansari - T-Oracle

                Hi Ed ,

                 

                Is your repository database and listener up and running here ?

                 

                 

                Confirm whether this is successful :

                 

                 

                NOTE:1448007.1 - 12c Cloud Control: How to Verify the Connectivity from 12c OMS to Repository Database using rcuJDBCEngine

                 

                 

                We would further require logs to review this issue .

                 

                 

                 

                Regards

                Ansari

                • 5. Re: OMS doesn't start - can't connect to database
                  EdSpn

                  Hi all,

                   

                  Many thanks for responding. I have opened an SR with Oracle 24 days ago. However, I just noticed I haven't heard from them for 13 days!

                  Must ping them to see what's happening...

                   

                  I will check out your suggestions. The db and the listener are all fine and up anyway.

                  After looking in a lot of logfiles (and uploading them to the SR), it looks as if it could have to do with credentials in a wallet:

                   

                  [oracle@ora-emcc log]$ cat secure.log

                  2013-10-15 20:07:36,265 [main] INFO oms.ListReposDetails logp.251 - Listing repository details from CredStore

                  2013-10-15 20:07:36,269 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.177 - Getting credentials from wallet

                  2013-10-15 20:07:36,728 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.192 - Read the credentials from wallet

                  2013-10-15 20:07:36,728 [main] INFO util.EmctlUtil logp.251 - Connecting over t3s to: ora-emcc.localdomain/7102 using id: weblogic

                  2013-10-15 20:07:38,639 [main] INFO mas.CredStoreUtil logp.251 - getCredential : Got creds for mapName -EM keyName - REPOS_DETAILS

                  2013-10-15 20:07:38,639 [main] INFO oms.ListReposDetails logp.251 - Exception thrown when getting repos details from CredStore: javax.management.RuntimeMBeanException: oracle.security.jps.service.keystore.KeyStoreServiceException: JPS-06516: Failed to get credential store. Reason oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException: PKI-02002: Unable to open the wallet. Check password.

                   

                  I will ping Oracle Support again (it's now with the security team) and see what the feedback is.

                   

                  Regards,
                  Ed

                  • 6. Re: OMS doesn't start - can't connect to database
                    mkathirv

                    I could see the SR is being checked by internal team.
                    In the meantime you can try this and let us know the results.

                     

                    Stop the OMS:
                    $OMS_HOME/bin/emctl stop oms -all -force

                    Make sure to kill the process running in the background.
                    $ ps -ef |grep EMGC
                    $ kill -9 <PID of EMGC>

                     

                    Filesystem where the OMS is installed do the following,

                    unmount & mount with no lock option

                    Restart OMS/Repo DB in below order.

                    Shutdown repository database
                    Stop listener
                    Start listener
                    Startup repository database
                    Start OMS > emctl start oms

                     

                    HTH
                    Mani

                    • 7. Re: OMS doesn't start - can't connect to database
                      EdSpn

                      Hi Mani,

                       

                      Thanks for the feedback. I just received an update from Oracle on the SR with some more instructions on different things to try/execute.

                      I ran into issues with Weblogic in step1 and couldn't understand the multitude of possible causes. After some Googling I found other users that ended up simply rebooting which did the trick. This worked for me as well as far as i can see now. I will do some tests to verify, but so far so good.

                       

                      Thanks,

                      Ed

                      • 8. Re: OMS doesn't start - can't connect to database
                        sxkumar

                        What is the SR number?

                        • 9. Re: OMS doesn't start - can't connect to database
                          EdSpn

                          Hi,

                           

                          It's SR 3-7938709991 : OEM 12c: OMS doesn't start - can't connect to database

                           

                          Regards,

                          Ed