Forum Stats

  • 3,838,768 Users
  • 2,262,398 Discussions
  • 7,900,755 Comments

Discussions

OMS doesn't start - can't connect to database

EdSpn
EdSpn Member Posts: 78
edited Nov 4, 2013 12:11PM in Enterprise Manager

Hi all,

My dev/test environment won't start OMS anymore. This really has us stopped in our tracks...

It will start the WebTier, but OMS won't come up:

[[email protected] log]$ $OMS_HOME/bin/emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.

Trying to check oms details results in an error as well:

[[email protected] config]$ $OMS_HOME/bin/emctl status oms -details -sysman_pwd $SYSMAN_PWD
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Exception in thread "main" java.util.MissingResourceException: Can't find bundle for base name oracle.security.jps.internal.common.resources.keystore.KeyStoreResources, locale
        at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1427)
        at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1250)
        at java.util.ResourceBundle.getBundle(ResourceBundle.java:952)
        at oracle.security.jps.util.JpsBundle.<init>(JpsBundle.java:70)
        at oracle.security.jps.util.JpsBundle.getResourceBundle(JpsBundle.java:151)
        at oracle.security.jps.util.JpsBundle.getResourceBundle(JpsBundle.java:106)
        at oracle.security.jps.JpsLocalizableException.getResourceBundle(JpsLocalizableException.java:157)
        at oracle.security.jps.JpsLocalizableException.getLocalizedMessage(JpsLocalizableException.java:140)
        at oracle.security.jps.JpsLocalizableException.getLocalizedMessage(JpsLocalizableException.java:131)
        at java.lang.Throwable.toString(Throwable.java:343)
        at java.lang.String.valueOf(String.java:2826)
        at java.lang.StringBuilder.append(StringBuilder.java:115)
        at java.lang.Throwable.printStackTraceAsCause(Throwable.java:536)
        at java.lang.Throwable.printStackTraceAsCause(Throwable.java:545)
        at java.lang.Throwable.printStackTraceAsCause(Throwable.java:545)
        at java.lang.Throwable.printStackTrace(Throwable.java:516)
        at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:59)
        at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:342)
        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:304)
        at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:234)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
        at org.apache.log4j.Category.callAppenders(Category.java:203)
        at org.apache.log4j.Category.forcedLog(Category.java:388)
        at org.apache.log4j.Category.error(Category.java:319)
        at oracle.sysman.emctl.oms.PrintOMSSetupInfo.main(PrintOMSSetupInfo.java:240)

In the WebLogic Admin Console, I can see that BIP, EMGC_ADMINSERVER and EMGC_OMS1 are all in state Running and health Ok.

Our OMS and OMR servers are (and have always been) 2 different machines. The database on OMR has always been (and is currently) up and available.

Checking that I can connect to the repository database on OMR from the OMS server:

[[email protected] ~]$ sqlplus sysman/[email protected]:1521/emccdb


SQL*Plus: Release 11.1.0.7.0 - Production on Thu Oct 10 23:34:53 2013


Copyright (c) 1982, 2008, Oracle.  All rights reserved.




Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options


SQL>

Any suggestions will be much appreciated.

Ed

EdSpn

Best Answer

  • EdSpn
    EdSpn Member Posts: 78
    Answer ✓

    Hi Mani,

    Thanks for the feedback. I just received an update from Oracle on the SR with some more instructions on different things to try/execute.

    I ran into issues with Weblogic in step1 and couldn't understand the multitude of possible causes. After some Googling I found other users that ended up simply rebooting which did the trick. This worked for me as well as far as i can see now. I will do some tests to verify, but so far so good.

    Thanks,

    Ed

Answers

  • Pls take a a look at emoms.trc, emoms.log and EMGC_OMS1.out

    One of these should have more details.

  • The below two files should give the exact root cause for the issue

    One of the known issue is if the Repository database archive log file system is full , and ORA-00257: archiver error is recorded in the below files

    ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out

    ../gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_ADMINSERVER.out

    Also Repository DB alert.log file should also be reviewed

    If Repos DB is clean then we can check the ../gc_inst/em/EMGC_OMS1/sysman/log/* for any security/wallet related issue

    You can also open a SR with EM team if the above does not help

    EdSpn
  • Bip
    Bip Member Posts: 33

    The 'emkey' may not be working currently in your environment.

    Run the command "emctl status emkey" from location <12c_OMS_HOME>/bin/

    If there is a issue with 'emkey' then you should be seeing an error as below,

    "The EMKey is not configured properly or is corrupted in the credential store and does not exist in the Management Repository"

    A quick solution in the above scenario would be to restore the 'emkey.ora' file from your file system backup if you have. The 'emkey' location in 12c OMS is <12c_OMS_HOME>/sysman/config/.

    Once you restore the file from your back up you will need to configure the emkey by running the below command,

    <12c_OMS_HOME>/bin/emctl config emkey -copy_to_credstore_from_file

    Let me know if this helps.

    Regards,

    Biplab

    EdSpn
  • EliazAnsari-Oracle
    EliazAnsari-Oracle Member Posts: 252 Employee

    Hi Ed ,

    Is your repository database and listener up and running here ?

    Confirm whether this is successful :

    NOTE:1448007.1 - 12c Cloud Control: How to Verify the Connectivity from 12c OMS to Repository Database using rcuJDBCEngine

    We would further require logs to review this issue .

    Regards

    Ansari

    EdSpn
  • EdSpn
    EdSpn Member Posts: 78

    Hi all,

    Many thanks for responding. I have opened an SR with Oracle 24 days ago. However, I just noticed I haven't heard from them for 13 days!

    Must ping them to see what's happening...

    I will check out your suggestions. The db and the listener are all fine and up anyway.

    After looking in a lot of logfiles (and uploading them to the SR), it looks as if it could have to do with credentials in a wallet:

    [[email protected] log]$ cat secure.log

    2013-10-15 20:07:36,265 [main] INFO oms.ListReposDetails logp.251 - Listing repository details from CredStore

    2013-10-15 20:07:36,269 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.177 - Getting credentials from wallet

    2013-10-15 20:07:36,728 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.192 - Read the credentials from wallet

    2013-10-15 20:07:36,728 [main] INFO util.EmctlUtil logp.251 - Connecting over t3s to: ora-emcc.localdomain/7102 using id: weblogic

    2013-10-15 20:07:38,639 [main] INFO mas.CredStoreUtil logp.251 - getCredential : Got creds for mapName -EM keyName - REPOS_DETAILS

    2013-10-15 20:07:38,639 [main] INFO oms.ListReposDetails logp.251 - Exception thrown when getting repos details from CredStore: javax.management.RuntimeMBeanException: oracle.security.jps.service.keystore.KeyStoreServiceException: JPS-06516: Failed to get credential store. Reason oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException: PKI-02002: Unable to open the wallet. Check password.

    I will ping Oracle Support again (it's now with the security team) and see what the feedback is.

    Regards,
    Ed

  • mkathirv
    mkathirv Member Posts: 64
    edited Nov 4, 2013 8:38AM

    I could see the SR is being checked by internal team.
    In the meantime you can try this and let us know the results.

    Stop the OMS:
    $OMS_HOME/bin/emctl stop oms -all -force

    Make sure to kill the process running in the background.
    $ ps -ef |grep EMGC
    $ kill -9 <PID of EMGC>

    Filesystem where the OMS is installed do the following,

    unmount & mount with no lock option

    Restart OMS/Repo DB in below order.

    Shutdown repository database
    Stop listener
    Start listener
    Startup repository database
    Start OMS > emctl start oms

    HTH
    Mani

    mkathirv
  • EdSpn
    EdSpn Member Posts: 78
    Answer ✓

    Hi Mani,

    Thanks for the feedback. I just received an update from Oracle on the SR with some more instructions on different things to try/execute.

    I ran into issues with Weblogic in step1 and couldn't understand the multitude of possible causes. After some Googling I found other users that ended up simply rebooting which did the trick. This worked for me as well as far as i can see now. I will do some tests to verify, but so far so good.

    Thanks,

    Ed

  • sxkumar
    sxkumar Member Posts: 161

    What is the SR number?

  • EdSpn
    EdSpn Member Posts: 78

    Hi,

    It's SR 3-7938709991 : OEM 12c: OMS doesn't start - can't connect to database

    Regards,

    Ed

This discussion has been closed.