7 Replies Latest reply on Dec 20, 2010 10:15 PM by user12050582

    Trying to install Grid Control 11.0.1 on Linux x86-64.... and failing

    Toaduser
      Is there anyone who succeeded? I'm sure there is....

      I've tried three times so far and I'm getting the same error.....

      After fumbling over all the prerequisites, it seems to be installing until "OMS Configuration" part, and it failed with following error in the log file.

      FINE: weblogic.nodemanager.NMConnectException: Connection refused. Could not connect to NodeManager. Check that it is running at jdisorc2:7403.
      Sep 29, 2010 3:56:39 PM oracle.sysman.omsca.util.ProcessOutputReader run
      FINE: at java.net.PlainSocketImpl.socketConnect(Native Method)

      ....

      Then more errors like

      FINE: NMProcess: Sep 29, 2010 3:56:49 PM weblogic.nodemanager.server.Handler handleStart
      Sep 29, 2010 3:56:49 PM oracle.sysman.omsca.util.ProcessOutputReader run
      FINE: NMProcess: WARNING: Exception while starting server 'EMGC_ADMINSERVER'
      Sep 29, 2010 3:56:49 PM oracle.sysman.omsca.util.ProcessOutputReader run
      FINE: NMProcess: java.lang.OutOfMemoryError: fork() failed due to insufficient memory


      But there are plenty of memory in the box.

      Also
      ...
      FINE: javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://jdisorc2:7101: Destination unreachable; nested exception is:
      Sep 29, 2010 3:56:51 PM oracle.sysman.omsca.util.ProcessOutputReader run
      ...

      NMProcess: INFO: Server output log file is '/u01/Oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out'
      NMProcess: <Sep 29, 2010 3:56:49 PM> <WARNING> <Exception while starting server 'EMGC_ADMINSERVER'>
      NMProcess: java.lang.OutOfMemoryError: fork() failed due to insufficient memory
      NMProcess:      at weblogic.nodemanager.util.UnixProcessControl.createProcess0(Native Method)
      NMProcess:      at weblogic.nodemanager.util.UnixProcessControl.createProcess(UnixProcessControl.java:90)
      NMProcess:      at weblogic.nodemanager.server.WLSProcessNativeImpl.start(WLSProcessNativeImpl.java:42)
      NMProcess:      at weblogic.nodemanager.server.WLSProcess.startProcess(WLSProcess.java:194)
      NMProcess:      at weblogic.nodemanager.server.ServerMonitor.startWLSProcess_inner(ServerMonitor.java:398)
      NMProcess:      at weblogic.nodemanager.server.ServerMonitor.startWLSProcess(ServerMonitor.java:336)
      NMProcess:      at weblogic.nodemanager.server.ServerMonitor.start(ServerMonitor.java:93)
      NMProcess:      at weblogic.nodemanager.server.ServerManager.startServer(ServerManager.java:372)
      NMProcess:      at weblogic.nodemanager.server.ServerManager.start(ServerManager.java:316)
      NMProcess:      at weblogic.nodemanager.server.Handler.handleStart(Handler.java:567)
      NMProcess:      at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:116)
      NMProcess:      at weblogic.nodemanager.server.Handler.run(Handler.java:70)
      NMProcess:      at java.lang.Thread.run(Thread.java:619)

      ...

      javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://jdisorc2:7101: Destination unreachable; nested exception is:
              java.net.ConnectException: Connection refused; No available router to destination]

      ...

      Then finally died after spooling out

      SEVERE: ERROR: Exception:
      oracle.sysman.omsca.framework.ConfigAssistantException: java.io.IOException: Cannot run program "/u01/Oracle/Middleware/oracle_common/common/bin/wlst.sh": java.io.IOException: error=12, Cannot allocate memory
      at oracle.sysman.omsca.util.CoreOMSConfigAssistantUtil.execCommand(CoreOMSConfigAssistantUtil.java:1885)
      at oracle.sysman.omsca.adapter.wls.OMSWLSAdapter.execCmd(OMSWLSAdapter.java:2525)
      at oracle.sysman.omsca.adapter.wls.OMSWLSAdapter.adapterDeleteOMS(OMSWLSAdapter.java:1577)
      at oracle.sysman.omsca.adapter.wls.OMSWLSAdapter.adapterCleanup(OMSWLSAdapter.java:2681)
      at oracle.sysman.omsca.adapter.wls.OMSWLSAdapter.adapterCreateInfra(OMSWLSAdapter.java:1362)
      at oracle.sysman.omsca.framework.OMSGenericAdapter.createInfrastructureEM(OMSGenericAdapter.java:108)
      at oracle.sysman.omsca.framework.OMSCAFreshInstall.execute(OMSCAFreshInstall.java:123)
      at oracle.sysman.omsca.framework.OMSConfigAssistantDriver.main(OMSConfigAssistantDriver.java:210)
      Caused by: java.io.IOException: Cannot run program "/u01/Oracle/Middleware/oracle_common/common/bin/wlst.sh": java.io.IOException: error=12, Cannot allocate memory
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
      at java.lang.Runtime.exec(Runtime.java:593)
      at java.lang.Runtime.exec(Runtime.java:431)
      at java.lang.Runtime.exec(Runtime.java:369)
      at oracle.sysman.omsca.util.CoreOMSConfigAssistantUtil.execCommand(CoreOMSConfigAssistantUtil.java:1845)
      ... 7 more
      Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
      at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
      at java.lang.ProcessImpl.start(ProcessImpl.java:65)
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
      ... 11 more
      Sep 29, 2010 3:58:36 PM oracle.sysman.omsca.framework.OMSConfigAssistantDriver main

      I tried to "retry" the process, and this time I got the following error:

      Sep 29, 2010 4:01:50 PM oracle.sysman.omsca.util.CoreOMSConfigAssistantUtil checkRepository
      SEVERE: Exception:
      java.sql.SQLRecoverableException: ORA-01034: ORACLE not available
      ORA-27102: out of memory
      Linux-x86_64 Error: 12: Cannot allocate memory
      Additional information: 1
      Additional information: 6586376
      Additional information: 8

      Any idea would be helpful...
        • 1. Re: Trying to install Grid Control 11.0.1 on Linux x86-64.... and failing
          Bazza
          How much memory in the box? How much is being used by other things? Have you set the kernel parameters (espcially thos to do with memory) up correctly? Have you configured the oracle user according to the documentation?

          Sorry to ask, but you the same error several times over.
          • 2. Re: Trying to install Grid Control 11.0.1 on Linux x86-64.... and failing
            Toaduser
            I finally opened a SR, and following the instruction from a tech support. I hope it works this time...
            I'm crossing my fingers...
            • 3. Re: Trying to install Grid Control 11.0.1 on Linux x86-64.... and failing
              rkthoka
              I am facing same issue.. please share if you have any information.

              Thanks,
              Krishna.
              • 4. Re: Trying to install Grid Control 11.0.1 on Linux x86-64.... and failing
                743319
                Which version of WLS is running? You have to have 10.3.2 for OMS to install successfully. I have OMS 11g running on RHEL 5.4 32 bit once I installed WLS 10.3.2. Hope this helps
                • 5. Re: Trying to install Grid Control 11.0.1 on Linux x86-64.... and failing
                  rkthoka
                  Yes I have wls-1032 only and quite interestingly am seeing wrong port(7,403 --> i do not know why) number in the log file

                  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                  emrep 03:36 PM stgrd101:/oracle/GRD/Middleware/wlserver_10.3/inventory/ContentsXML > grep -i version /oracle/GRD/Middleware/registry.xml
                  <?xml version="1.0" encoding="UTF-8"?>
                  <component name="Common Infrastructure Engineering" version="6.8.0.0" InstallDir="">
                  <component name="WebLogic Server" version="10.3.2.0" InstallDir="/oracle/GRD/Middleware/wlserver_10.3">
                  <component name="Oracle Configuration Manager" version="10.3.1.2" InstallDir="">
                  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                  ______________________________________________________________________________________________
                  /oracle/GRD/Middleware/oms11g/cfgtoollogs/omsca/omsca_20101021140122.log
                  ______________________________________________________________________________________________
                  Code of Exception: Error occured while performing nmConnect : Cannot connect to Node Manager. : Connection refused. Could not connect to NodeManager. Check that it is running at stgrd101.frxntnyc.frx2.com:7,403.
                  Use dumpStack() to view the full stacktrace
                  <traceback object at 2>
                  ______________________________________________________________________________________________


                  Thanks,
                  Krishna.

                  Edited by: 795338 on Oct 21, 2010 1:49 PM
                  • 6. Re: Trying to install Grid Control 11.0.1 on Linux x86-64.... and failing
                    rkthoka
                    My issue was HOST file has unwanted information.

                    Some how other cluster IP' including SCAN,VIP,private and public IP been there. For some reason NODE MANGER thinking that Web Logic and OMS installing server is cluster server and it is trying to look for more information when Node Manger trying establish connect. (Atleast my understanding)

                    so I have cleaned up /etc/hosts file and retry it worked fine.

                    Remebere when you hit retry it mostly likely fail immediately, because (ps -ef|grep wls) weblogic process already runs, when you hit retry it try restart and then fails with message WLS already up and running. So kill or shutdown the exisitng WLS


                    I followed these steps:
                    0) clean host file

                    1) ps -ef|grep -i wls

                    2) kill -9 <PROCESS_ID>

                    3) cd $MW_HOME/wlserver_10.3/common/bin

                    4) vi wlst.sh
                    and edit this set -x
                    echo "before sleep"
                    sleep 200 # sleep for 3 plus minutes
                    echo "after sleep"


                    I need to do this because when you restart the OMS config, installer recreating another folder called "emnodemanager<DATA_TIME_STAMP> some thing like below and it does not any files (nodemanger.properties etc)
                    And remember this directory creates at run time, so before WLS process starts (wlst.sh) which starts nodemanger tool.

                    5) on OUI, hit retry

                    6) tail -f $MW_HOME/oms11g/cfgtoollogs/omsca/omsca_<DATE_TIME_STAMP>.log

                    when you see "before sleep" in the log file

                    7) cd $MW_HOME/wlserver_10.3/common

                    8) ls -ltr emnode* and see latest folder "cd to latest emnodemanger<DATE_TIME_STAMP> folder"

                    9) cp $MW_HOME/wlserver_10.3/common/emnodemanager/* .

                    10) Monitor the $MW_HOME/oms11g/cfgtoollogs/omsca/omsca_<DATE_TIME_STAMP>.log


                    whether you have the /etc/host file issue or some thing else, if you retrying OMS config this is the same process ....

                    Finally this is how I fix it.. but you may also find some other way to bypass this

                    Thanks,
                    Krishna.
                    • 7. Re: Trying to install Grid Control 11.0.1 on Linux x86-64.... and failing
                      user12050582
                      In my case, I resolved it by doing the below after struggling for 3 full days:


                      Error occured while performing nmConnect : Cannot connect to Node Manager. : Connection refused. Could not connect to NodeManager. Check that it is running at hostname.domain.com:7,403.

                      Port used in /wlserver_10.3/common/emnodemanager/nodemanager.properties
                      was 7403 as stated in the message

                      Port used in
                      vi ./oms11g/cfgtoollogs/omsca/logs20101220201721/ConfigDir/config.xml
                      was 7401

                      If it's different, then it won't work. Make is same in config.xml to 7401, retry again, now it worked.

                      Thanks,
                      Shakti Bhadupotey