This discussion is archived
9 Replies Latest reply: Nov 4, 2013 9:11 AM by EdSpn RSS

OMS doesn't start - can't connect to database

EdSpn Newbie
Currently Being Moderated

Hi all,

 

My dev/test environment won't start OMS anymore. This really has us stopped in our tracks...

It will start the WebTier, but OMS won't come up:

 

[oracle@ora-emcc log]$ $OMS_HOME/bin/emctl status oms

Oracle Enterprise Manager Cloud Control 12c Release 2

Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.

WebTier is Up

Oracle Management Server is not functioning because of the following reason:

Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.

 

Trying to check oms details results in an error as well:

[oracle@ora-emcc config]$ $OMS_HOME/bin/emctl status oms -details -sysman_pwd $SYSMAN_PWD

Oracle Enterprise Manager Cloud Control 12c Release 2

Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.

Exception in thread "main" java.util.MissingResourceException: Can't find bundle for base name oracle.security.jps.internal.common.resources.keystore.KeyStoreResources, locale

        at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1427)

        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1250)

        at java.util.ResourceBundle.getBundle(ResourceBundle.java:952)

        at oracle.security.jps.util.JpsBundle.<init>(JpsBundle.java:70)

        at oracle.security.jps.util.JpsBundle.getResourceBundle(JpsBundle.java:151)

        at oracle.security.jps.util.JpsBundle.getResourceBundle(JpsBundle.java:106)

        at oracle.security.jps.JpsLocalizableException.getResourceBundle(JpsLocalizableException.java:157)

        at oracle.security.jps.JpsLocalizableException.getLocalizedMessage(JpsLocalizableException.java:140)

        at oracle.security.jps.JpsLocalizableException.getLocalizedMessage(JpsLocalizableException.java:131)

        at java.lang.Throwable.toString(Throwable.java:343)

        at java.lang.String.valueOf(String.java:2826)

        at java.lang.StringBuilder.append(StringBuilder.java:115)

        at java.lang.Throwable.printStackTraceAsCause(Throwable.java:536)

        at java.lang.Throwable.printStackTraceAsCause(Throwable.java:545)

        at java.lang.Throwable.printStackTraceAsCause(Throwable.java:545)

        at java.lang.Throwable.printStackTrace(Throwable.java:516)

        at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:59)

        at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:342)

        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:304)

        at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:234)

        at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)

        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)

        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)

        at org.apache.log4j.Category.callAppenders(Category.java:203)

        at org.apache.log4j.Category.forcedLog(Category.java:388)

        at org.apache.log4j.Category.error(Category.java:319)

        at oracle.sysman.emctl.oms.PrintOMSSetupInfo.main(PrintOMSSetupInfo.java:240)

 

In the WebLogic Admin Console, I can see that BIP, EMGC_ADMINSERVER and EMGC_OMS1 are all in state Running and health Ok.

 

Our OMS and OMR servers are (and have always been) 2 different machines. The database on OMR has always been (and is currently) up and available.

Checking that I can connect to the repository database on OMR from the OMS server:

 

[oracle@ora-emcc ~]$ sqlplus sysman/$SYSMAN_PWD@ora-emccdb:1521/emccdb

 

 

SQL*Plus: Release 11.1.0.7.0 - Production on Thu Oct 10 23:34:53 2013

 

 

Copyright (c) 1982, 2008, Oracle.  All rights reserved.

 

 

 

 

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

 

 

SQL>

 

 

Any suggestions will be much appreciated.

 

Ed

  • 1. Re: OMS doesn't start - can't connect to database
    user729207 Newbie
    Currently Being Moderated

    Pls take a a look at emoms.trc, emoms.log and EMGC_OMS1.out

    One of these should have more details.

  • 2. Re: OMS doesn't start - can't connect to database
    Rahul-EM Journeyer
    Currently Being Moderated

    The below two files should give the exact root cause for the issue

    One of the known issue is if the Repository database archive log file system is full , and ORA-00257: archiver error is recorded in the below files

    ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out

    ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_ADMINSERVER.out

    Also Repository DB alert.log file should also be reviewed

     

    If Repos DB is clean then we can check the ../gc_inst/em/EMGC_OMS1/sysman/log/* for any security/wallet related issue

     

    You can also open a SR with EM team if the above does not help

  • 3. Re: OMS doesn't start - can't connect to database
    Bip Explorer
    Currently Being Moderated

    The 'emkey' may not be working currently in your environment.

     

    Run the command "emctl status emkey" from location <12c_OMS_HOME>/bin/

     

    If there is a issue with 'emkey' then you should be seeing an error as below,

     

    "The EMKey is not configured properly or is corrupted in the credential store and does not exist in the Management Repository"

     

    A quick solution in the above scenario would be to restore the 'emkey.ora' file from your file system backup if you have. The 'emkey' location in 12c OMS is <12c_OMS_HOME>/sysman/config/.

     

    Once you restore the file from your back up you will need to configure the emkey by running the below command,

     

    <12c_OMS_HOME>/bin/emctl config emkey -copy_to_credstore_from_file

     

    Let me know if this helps.

     

     

    Regards,

    Biplab

  • 4. Re: OMS doesn't start - can't connect to database
    Ansari - T Expert
    Currently Being Moderated

    Hi Ed ,

     

    Is your repository database and listener up and running here ?

     

     

    Confirm whether this is successful :

     

     

    NOTE:1448007.1 - 12c Cloud Control: How to Verify the Connectivity from 12c OMS to Repository Database using rcuJDBCEngine

     

     

    We would further require logs to review this issue .

     

     

     

    Regards

    Ansari

  • 5. Re: OMS doesn't start - can't connect to database
    EdSpn Newbie
    Currently Being Moderated

    Hi all,

     

    Many thanks for responding. I have opened an SR with Oracle 24 days ago. However, I just noticed I haven't heard from them for 13 days!

    Must ping them to see what's happening...

     

    I will check out your suggestions. The db and the listener are all fine and up anyway.

    After looking in a lot of logfiles (and uploading them to the SR), it looks as if it could have to do with credentials in a wallet:

     

    [oracle@ora-emcc log]$ cat secure.log

    2013-10-15 20:07:36,265 [main] INFO oms.ListReposDetails logp.251 - Listing repository details from CredStore

    2013-10-15 20:07:36,269 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.177 - Getting credentials from wallet

    2013-10-15 20:07:36,728 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.192 - Read the credentials from wallet

    2013-10-15 20:07:36,728 [main] INFO util.EmctlUtil logp.251 - Connecting over t3s to: ora-emcc.localdomain/7102 using id: weblogic

    2013-10-15 20:07:38,639 [main] INFO mas.CredStoreUtil logp.251 - getCredential : Got creds for mapName -EM keyName - REPOS_DETAILS

    2013-10-15 20:07:38,639 [main] INFO oms.ListReposDetails logp.251 - Exception thrown when getting repos details from CredStore: javax.management.RuntimeMBeanException: oracle.security.jps.service.keystore.KeyStoreServiceException: JPS-06516: Failed to get credential store. Reason oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException: PKI-02002: Unable to open the wallet. Check password.

     

    I will ping Oracle Support again (it's now with the security team) and see what the feedback is.

     

    Regards,
    Ed

  • 6. Re: OMS doesn't start - can't connect to database
    mkathirv Expert
    Currently Being Moderated

    I could see the SR is being checked by internal team.
    In the meantime you can try this and let us know the results.

     

    Stop the OMS:
    $OMS_HOME/bin/emctl stop oms -all -force

    Make sure to kill the process running in the background.
    $ ps -ef |grep EMGC
    $ kill -9 <PID of EMGC>

     

    Filesystem where the OMS is installed do the following,

    unmount & mount with no lock option

    Restart OMS/Repo DB in below order.

    Shutdown repository database
    Stop listener
    Start listener
    Startup repository database
    Start OMS > emctl start oms

     

    HTH
    Mani

  • 7. Re: OMS doesn't start - can't connect to database
    EdSpn Newbie
    Currently Being Moderated

    Hi Mani,

     

    Thanks for the feedback. I just received an update from Oracle on the SR with some more instructions on different things to try/execute.

    I ran into issues with Weblogic in step1 and couldn't understand the multitude of possible causes. After some Googling I found other users that ended up simply rebooting which did the trick. This worked for me as well as far as i can see now. I will do some tests to verify, but so far so good.

     

    Thanks,

    Ed

  • 8. Re: OMS doesn't start - can't connect to database
    sxkumar Explorer
    Currently Being Moderated

    What is the SR number?

  • 9. Re: OMS doesn't start - can't connect to database
    EdSpn Newbie
    Currently Being Moderated

    Hi,

     

    It's SR 3-7938709991 : OEM 12c: OMS doesn't start - can't connect to database

     

    Regards,

    Ed

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points