12 Replies Latest reply: Feb 3, 2012 8:52 AM by Nayas RSS

    ebs r12 oacore memory

    Nayas
      Hi All,

      Ebs r12.1.3
      rdbms 11.2.0.2

      I would like to know if there is a way to monitor oacore memory usage and if there is a way to alert us when the memory is close to the maximum memory available (-Xmx parameter)?
      I also would like to know, our oacore seems to fill during the week and at one point, doesn't have enough memory and they froze/crash. We have to bounce them to fix the issue.
      Does any of you do on a regular basic a bounce of OC4J components (adopmnctl.sh stop / start) ?

      Thank you,
      Felix
        • 1. Re: ebs r12 oacore memory
          EBSDBA
          Hi,
          Please see if it helps

          http://riktutorials.blogspot.com/2010/04/how-to-monitor-memory-usage-in-oc4j-as.html

          Simply source the appl. env file and run jconsole (using vnc or any other graphical tool).
          It will list you the local processes.
          To find the oacore process, do the following
          applmgr$ ps -ef|grep oacorestd.err
          It'll show you the PID, The same pid you can see it jconsole.

          More information:

          How to create a Java heapdump on e-Business Suite ? [ID 835909.1]
          Guidelines to setup the JVM in Apps Ebusiness Suite 11i and R12 [ID 362851.1]

          http://blogs.oracle.com/stevenChan/entry/using_jconsole_to_monitor_apps
          OACore monitoring question (A previous thread of yourself :) )

          Thanks
          • 2. Re: ebs r12 oacore memory
            Nayas
            Hi,

            Oh, I forgot I already asked the same question :S
            I am trying to use jconsole but in the local tab, I do not have any oacore process, only GSM.

            I am getting the PID from $INST_TOP/admin/scripts/adopmnctl.sh status.
            When I try to do jconsole PID, I am getting:
            [server.domain.com]:> $INST_TOP/admin/scripts/adopmnctl.sh status
            
            You are running adopmnctl.sh version 120.6.12010000.5
            
            Checking status of OPMN managed processes...
            
            Processes in Instance: SID_server.server.domain.com
            ---------------------------------+--------------------+---------+---------
            ias-component                    | process-type       |     pid | status
            ---------------------------------+--------------------+---------+---------
            OC4JGroup:default_group          | OC4J:oafm          | 2891892 | Alive
            OC4JGroup:default_group          | OC4J:oacore        | 104695~ | Alive
            HTTP_Server                      | HTTP_Server        | 101213~ | Alive
            
            [server.domain.com]:> which jconsole
            $IAS_ORACLE_HOME/10.1.3/appsutil/jdk/bin/jconsole
            
            [server.domain.com]:> jconsole 10469588
            com.ibm.tools.attach.AttachNotSupportedException: target not found
                    at com.ibm.tools.attach.javaSE.VirtualMachineImpl.tryAttachTarget(VirtualMachineImpl.java:303)
                    at com.ibm.tools.attach.javaSE.VirtualMachineImpl.attachTarget(VirtualMachineImpl.java:86)
                    at com.ibm.tools.attach.javaSE.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:37)
                    at ibm.tools.attach.J9AttachProvider.attachVirtualMachine(J9AttachProvider.java:60)
                    at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:221)
                    at sun.tools.jconsole.LocalVirtualMachine.getLocalVirtualMachine(LocalVirtualMachine.java:205)
                    at sun.tools.jconsole.JConsole.main(JConsole.java:924)
            Invalid process id:10469588
            But:
            [server.domain.com]:> ps -ef | grep 104695
             applmgr 9469982 10244248   0 15:23:29  pts/2  0:00 grep 104695
             applmgr 10469588  1806434   0 14:18:34      -  0:13 $IAS_ORACLE_HOME/10.1.3/appsutil/jdk/bin/java -DCLIENT_PROCESSID=10469588 -verbose:gc -Xmx512M -Xms128M -Dcom.sun.management.jmxremote -Djava.security.policy=$IAS_ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=$IAS_ORACLE_HOME/j2ee/oacore/config/jazn.xml -Dhttp.cookie.ignoreCommaInCookiesNamed=X_NoMatchingCookies -Doracle.ons.oraclehome=$IAS_ORACLE_HOME -Doracle.home=$ISA_ORACLE_HOME -Doracle.ons.oracleconfighome=$INST_TOP/ora/10.1.3 -Doracle.ons.clustername=default -Doracle.ons.instancename=SID_server.server.domain.com -Dopmn.compatible=904 -Doracle.ons.indexid=oacore.default_group.1 -Doracle.ons.numprocs=1 -Doracle.ons.uid=745050216 -Doracle.oc4j.groupname=default_group -Doracle.oc4j.instancename=oacore -Doracle.oc4j.islandname=default_group -Doracle.opmn.routingid=g_rt_id -DOPMN=true -jar oc4j.jar -config $INST_TOP/ora/10.1.3/j2ee/oacore/config/server.xml -properties -out $INST_TOP/logs/ora/10.1.3/opmn/oacorestd.out -err $INST_TOP/logs/ora/10.1.3/opmn/oacorestd.err -ports default-web-site:ajp:21575,rmi:20075,jms:23075
            I do not understand why it says invalid process id.

            If I simply start jconsole without providing PID, I can see the local JVM. All I see is the FNDOPP / FNDCPGSC processes, nothing else.

            Any idea?

            Thank you,
            Felix
            • 3. Re: ebs r12 oacore memory
              EBSDBA
              Hi Feix,
              Just type jconsole

              ok, you will list of processes right.

              1. do

              $ps -ef|grep oacorestd.err
              IT shows you the process id for oacore.

              Match that process in the list of jconsole for oacore

              2.

              or you can put the cursor on each of the processes, it shows you the complete path (helper) . Like oacore,forms etc.

              Thanks
              • 4. Re: ebs r12 oacore memory
                Nayas
                Hi,

                Problem with simply typing jconsole is that my oacore processes are not displayed in the local tab. Only FNDOPP and FNDGSM.

                Thanks
                • 5. Re: ebs r12 oacore memory
                  EBSDBA
                  Hi,
                  I checked it on one of my instances (12.1.3 with 11.2.0.2). It displays here fine here.

                  when you do ps -ef|grep oacorestd.err does it return any processes.

                  Thanks
                  • 6. Re: ebs r12 oacore memory
                    Nayas
                    Hi,

                    Note: in our tests instance, we only have 1 oacore process where as in prod, we have 2
                    [server.domain.com]:> ps -fu applmgr | grep oacorestd.err
                     applmgr 10469588  1806434   0 14:18:34      -  0:37 $IAS_ORACLE_HOME/appsutil/jdk/bin/java -DCLIENT_PROCESSID=10469588 -verbose:gc -Xmx512M -Xms128M -Dcom.sun.management.jmxremote -Djava.security.policy=$IAS_ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=$INST_TOP/ora/10.1.3/j2ee/oacore/config/jazn.xml -Dhttp.cookie.ignoreCommaInCookiesNamed=X_NoMatchingCookies -Doracle.ons.oraclehome=$IAS_ORACLE_HOME -Doracle.home=$IAS_ORACLE_HOME -Doracle.ons.oracleconfighome=$INST_TOP/ora/10.1.3 -Doracle.ons.clustername=default -Doracle.ons.instancename=IMGCPU_server.server.domain.com -Dopmn.compatible=904 -Doracle.ons.indexid=oacore.default_group.1 -Doracle.ons.numprocs=1 -Doracle.ons.uid=745050216 -Doracle.oc4j.groupname=default_group -Doracle.oc4j.instancename=oacore -Doracle.oc4j.islandname=default_group -Doracle.opmn.routingid=g_rt_id -DOPMN=true -jar oc4j.jar -config $INST_TOP/ora/10.1.3/j2ee/oacore/config/server.xml -properties -out $INST_TOP/logs/ora/10.1.3/opmn/oacorestd.out -err $INST_TOP/logs/ora/10.1.3/opmn/oacorestd.err -ports default-web-site:ajp:21575,rmi:20075,jms:23075
                    Now if I do
                    [server.domain.com]:> jconsole 1806434
                    com.ibm.tools.attach.AttachNotSupportedException: target not found
                            at com.ibm.tools.attach.javaSE.VirtualMachineImpl.tryAttachTarget(VirtualMachineImpl.java:303)
                            at com.ibm.tools.attach.javaSE.VirtualMachineImpl.attachTarget(VirtualMachineImpl.java:86)
                            at com.ibm.tools.attach.javaSE.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:37)
                            at ibm.tools.attach.J9AttachProvider.attachVirtualMachine(J9AttachProvider.java:60)
                            at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:221)
                            at sun.tools.jconsole.LocalVirtualMachine.getLocalVirtualMachine(LocalVirtualMachine.java:205)
                            at sun.tools.jconsole.JConsole.main(JConsole.java:924)
                    Invalid process id:1806434
                    If I simply type jconsole, in Java Monitoring & Management Console, I do not have my pid 1806434...

                    Any idea?

                    Thank you,
                    Felix
                    • 7. Re: ebs r12 oacore memory
                      EBSDBA
                      Hi Felix,
                      applmgr 10469588 1806434
                      Sorry the pid is 10469588 not 1806434

                      Thanks
                      • 8. Re: ebs r12 oacore memory
                        Nayas
                        Hi,

                        Sorry, my mistake.

                        Same thing with the PID and not the PPID:
                        [server.domain.com]:> jconsole 10469588
                        com.ibm.tools.attach.AttachNotSupportedException: target not found
                                at com.ibm.tools.attach.javaSE.VirtualMachineImpl.tryAttachTarget(VirtualMachineImpl.java:303)
                                at com.ibm.tools.attach.javaSE.VirtualMachineImpl.attachTarget(VirtualMachineImpl.java:86)
                                at com.ibm.tools.attach.javaSE.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:37)
                                at ibm.tools.attach.J9AttachProvider.attachVirtualMachine(J9AttachProvider.java:60)
                                at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:221)
                                at sun.tools.jconsole.LocalVirtualMachine.getLocalVirtualMachine(LocalVirtualMachine.java:205)
                                at sun.tools.jconsole.JConsole.main(JConsole.java:924)
                        Invalid process id:10469588
                        Thanks
                        • 9. Re: ebs r12 oacore memory
                          Nayas
                          Hi,

                          Strange, I tried to same on a R12.1.1 lab we have at work and it works correctly. I can connect no problem to the oacore with jconsole <oacore_pid> ...

                          I do not get why it doesn't work for that specific instance. Any special settings to allow jconsole to connect to oacore process?

                          Thanks
                          • 10. Re: ebs r12 oacore memory
                            Nayas
                            Hi

                            I start believing it's an issue with AIX JRE ... works greate on linux but not on AIX...
                            • 11. Re: ebs r12 oacore memory
                              EBSDBA
                              Hi,
                              Yeah you are right.

                              This could be a bug in AIX.

                              Thanks
                              • 12. Re: ebs r12 oacore memory
                                Nayas
                                Hi,
                                FYI - we fixed jconsole issue.

                                With AIX, JDK creates a folder under /tmp call .bom_ibm_tools_attach. Inside this folder, there are folders, each corresponding to the PID of jvm. The owner of the directory is the owner of the JVM.
                                Our issue with jconsole was that, we changed, in applmgr .profile the variable TMPDIR to another directory than /tmp (for example, let say we use appldev, /apps/ebs/DEV/devtmp).

                                When we go to that directory, there is also a .com_ibm_tools_attach directory with sub folders for each JVM running under appldev BESIDE oacore jvm. Oacore jvm is under /tmp.
                                So when doing "jconsole OACORE_PID" we are getting "Invalid process id" because it cannot find the directory /apps/ebs/DEV/devtmp/.com_ibm_tools/attach/OACORE_PID because it's under /tmp.
                                The workaround is to do TMPDIR=tmp jconsole OACORE_PID. Bingo, now it works.

                                Now, where is the variable that says oacore tmp folder goes to /tmp if we overwrite everything to go to /apps/ebs/DEV/devtmp ...?
                                We tried to variable in opmn.xml the following line but it didn't work:

                                From
                                <ias-instance id="SID_sever.server.domain.com" name="SID_sever.server.domain.com">
                                  <environment>
                                    <variable id="TMP" value="/tmp" />
                                  </environment>
                                To
                                <ias-instance id="SID_sever.server.domain.com" name="SID_sever.server.domain.com">
                                  <environment>
                                    <variable id="TMP" value="/apps/ebs/DEV/devtmp" />
                                  </environment>
                                But regardless of jconsole and our issue with jconsole, i cannot believe we are the only one trying to monitor oacore and alert us when oacore memory is close to the maximum allowed?
                                I cannot find anything to monitor and alert when memory gets close to the mamimum allowed...

                                Thank you,
                                Felix