1 Reply Latest reply: Nov 16, 2012 11:37 AM by 899771 RSS

    Oracle BPM 11.1.1.5 Performance issues

    899771
      Hi,

      I have 2 Node Cluster of Oracle SOA 11.1.1.5 installed. Have a separate cluster for SOA/BPM, BAM, OSB, WSM. Here are the OS and Java versions

      Java Vendor: HP
      Java Version: 1.6.0.14
      OS: HP-UX
      OS Version: B.11.31

      Running into an issue where Oracle BPM is performing very slow the task forms takes forever to come up and performing any actions takes too long to proceed som time it keeps timing out. I have over 100 SOA Process running plus some BPM Process but BMP Workspace application is running too slow as have customized tasks forms. But the same works a bit better in other instance which is non-clustered. Any idea what can be done on the server level to get around the performance issues. So far have modified Audit Levels and reduced soa-infra bpm log have the following settings in setDomainEnv.sh for USER_MEM_ARGS=-Xms2000m -Xmx6000m -XX:PermSize=1000m

      But not help. Any idea what else to look into to get around these bpm performance issues. Here is what I have in setSOADomainEnv.sh

      # 8395254: add -da:org.apache.xmlbeans... in EXTRA_JAVA_PROPERTIES
      EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -da:org.apache.xmlbeans..."
      
      XENGINE_DIR="${SOA_ORACLE_HOME}/soa/thirdparty/edifecs/XEngine"
      DEFAULT_MEM_ARGS="-Xms512m -Xmx1024m"
      PORT_MEM_ARGS="-Xms768m -Xmx1536m"
      
      if [ "${JAVA_VENDOR}" != "Oracle" ] ; then
        DEFAULT_MEM_ARGS="${DEFAULT_MEM_ARGS} -XX:PermSize=128m -XX:MaxPermSize=512m"
        PORT_MEM_ARGS="${PORT_MEM_ARGS} -XX:PermSize=256m -XX:MaxPermSize=512m"
      fi
      
      #========================================================
      # setup LD_LIBRARY_PATH if directory is present...
      #========================================================
      if [ -d ${XENGINE_DIR}/bin ]; then
         LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${XENGINE_DIR}/bin"
         export LD_LIBRARY_PATH
      fi
      
      #========================================================
      # setup platform specific environment variables
      #========================================================
      case ${PLATFORM_TYPE} in
        #-----------------------------------------------------
        # AIX
        #-----------------------------------------------------
        AIX)
      
          if [ -d ${XENGINE_DIR}/bin ]; then
             LIBPATH="${LIBPATH}:${XENGINE_DIR}/bin"
             export LIBPATH
          fi
      
          USER_MEM_ARGS=${PORT_MEM_ARGS}
          export USER_MEM_ARGS
      
          # Fix for 7828060
          POST_CLASSPATH=${POST_CLASSPATH}:${SOA_ORACLE_HOME}/soa/modules/soa-ibm-addon.jar
      
          # Fix for 7520915 and 8264518 and 8305217
          EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Djavax.xml.datatype.DatatypeFactory=org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl -Djava.endorsed.dirs=${SOA_ORACLE_HOME}/bam/modules/org.apache.xalan_2.7.1"
          export EXTRA_JAVA_PROPERTIES
      
          ;;
      
        #-----------------------------------------------------
        # HPUX
        #-----------------------------------------------------
        HP-UX)
      
          if [ -d ${XENGINE_DIR}/bin ]; then
             SHLIB_PATH="${SHLIB_PATH}:${XENGINE_DIR}/bin"
             export SHLIB_PATH
          fi
      
          LD_LIBRARY_PATH="${XENGINE_DIR}/bin:${LD_LIBRARY_PATH}"
          export LD_LIBRARY_PATH
      
          USER_MEM_ARGS="-d64 ${PORT_MEM_ARGS}"
          export USER_MEM_ARGS
      
          ;;
      And here is what I have in setDomainEnv.sh
      XMS_SUN_64BIT="256"
      export XMS_SUN_64BIT
      XMS_SUN_32BIT="256"
      export XMS_SUN_32BIT
      XMX_SUN_64BIT="512"
      export XMX_SUN_64BIT
      XMX_SUN_32BIT="512"
      export XMX_SUN_32BIT
      XMS_JROCKIT_64BIT="256"
      export XMS_JROCKIT_64BIT
      XMS_JROCKIT_32BIT="256"
      export XMS_JROCKIT_32BIT
      XMX_JROCKIT_64BIT="512"
      export XMX_JROCKIT_64BIT
      XMX_JROCKIT_32BIT="512"
      export XMX_JROCKIT_32BIT
      
      
      if [ "${JAVA_VENDOR}" = "Sun" ] ; then
           WLS_MEM_ARGS_64BIT="-Xms256m -Xmx512m"
           export WLS_MEM_ARGS_64BIT
           WLS_MEM_ARGS_32BIT="-Xms256m -Xmx512m"
           export WLS_MEM_ARGS_32BIT
      else
           WLS_MEM_ARGS_64BIT="-Xms512m -Xmx512m"
           export WLS_MEM_ARGS_64BIT
           WLS_MEM_ARGS_32BIT="-Xms512m -Xmx512m"
           export WLS_MEM_ARGS_32BIT
      fi
      
      if [ "${JAVA_VENDOR}" = "Oracle" ] ; then
           CUSTOM_MEM_ARGS_64BIT="-Xms${XMS_JROCKIT_64BIT}m -Xmx${XMX_JROCKIT_64BIT}m"
           export CUSTOM_MEM_ARGS_64BIT
           CUSTOM_MEM_ARGS_32BIT="-Xms${XMS_JROCKIT_32BIT}m -Xmx${XMX_JROCKIT_32BIT}m"
           export CUSTOM_MEM_ARGS_32BIT
      else
           CUSTOM_MEM_ARGS_64BIT="-Xms${XMS_SUN_64BIT}m -Xmx${XMX_SUN_64BIT}m"
           export CUSTOM_MEM_ARGS_64BIT
           CUSTOM_MEM_ARGS_32BIT="-Xms${XMS_SUN_32BIT}m -Xmx${XMX_SUN_32BIT}m"
           export CUSTOM_MEM_ARGS_32BIT
      fi
      
      
      MEM_ARGS_64BIT="${CUSTOM_MEM_ARGS_64BIT}"
      export MEM_ARGS_64BIT
      
      MEM_ARGS_32BIT="${CUSTOM_MEM_ARGS_32BIT}"
      export MEM_ARGS_32BIT
      
      if [ "${JAVA_USE_64BIT}" = "true" ] ; then
           MEM_ARGS="${MEM_ARGS_64BIT}"
           export MEM_ARGS
      else
           MEM_ARGS="${MEM_ARGS_32BIT}"
           export MEM_ARGS
      fi
      
      MEM_PERM_SIZE_64BIT="-XX:PermSize=128m"
      export MEM_PERM_SIZE_64BIT
      
      MEM_PERM_SIZE_32BIT="-XX:PermSize=128m"
      export MEM_PERM_SIZE_32BIT
      
      if [ "${JAVA_USE_64BIT}" = "true" ] ; then
           MEM_PERM_SIZE="${MEM_PERM_SIZE_64BIT}"
           export MEM_PERM_SIZE
      else
           MEM_PERM_SIZE="${MEM_PERM_SIZE_32BIT}"
           export MEM_PERM_SIZE
      fi
      
      MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=512m"
      export MEM_MAX_PERM_SIZE_64BIT
      
      MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=512m"
      export MEM_MAX_PERM_SIZE_32BIT
      
      if [ "${JAVA_USE_64BIT}" = "true" ] ; then
           MEM_MAX_PERM_SIZE="${MEM_MAX_PERM_SIZE_64BIT}"
           export MEM_MAX_PERM_SIZE
      else
           MEM_MAX_PERM_SIZE="${MEM_MAX_PERM_SIZE_32BIT}"
           export MEM_MAX_PERM_SIZE
      fi
      
      if [ "${JAVA_VENDOR}" = "Sun" ] ; then
           if [ "${PRODUCTION_MODE}" = "" ] ; then
                MEM_DEV_ARGS="-XX:CompileThreshold=8000 ${MEM_PERM_SIZE} "
                export MEM_DEV_ARGS
           fi
      fi
      
      # Had to have a separate test here BECAUSE of immediate variable expansion on windows
      
      if [ "${JAVA_VENDOR}" = "Sun" ] ; then
           MEM_ARGS="${MEM_ARGS} ${MEM_DEV_ARGS} ${MEM_MAX_PERM_SIZE}"
           export MEM_ARGS
      fi
      
      if [ "${JAVA_VENDOR}" = "HP" ] ; then
           MEM_ARGS="${MEM_ARGS} ${MEM_MAX_PERM_SIZE}"
           export MEM_ARGS
      fi
      
      if [ "${JAVA_VENDOR}" = "Apple" ] ; then
           MEM_ARGS="${MEM_ARGS} ${MEM_MAX_PERM_SIZE}"
           export MEM_ARGS
      fi
      
      if [ "${debugFlag}" = "true" ] ; then
           JAVA_OPTIONS="${JAVA_OPTIONS} -da:org.apache.xmlbeans... "
           export JAVA_OPTIONS
      fi
      
      export USER_MEM_ARGS="-Xms4g -Xmx6g -XX:PermSize=2g -XX:+UseParallelGC -XX:+UseParallelOldGC"
      Here is the output of the top command
      Load averages: 0.08, 0.06, 0.06
      315 processes: 205 sleeping, 110 running
      Cpu states:
      CPU   LOAD   USER   NICE    SYS   IDLE  BLOCK  SWAIT   INTR   SSYS
       0    0.06   4.0%   0.2%   1.0%  94.8%   0.0%   0.0%   0.0%   0.0%
       2    0.07   3.6%   5.0%   0.2%  91.2%   0.0%   0.0%   0.0%   0.0%
       4    0.07   2.0%   0.2%   0.0%  97.8%   0.0%   0.0%   0.0%   0.0%
       6    0.07   2.4%   0.2%   0.8%  96.6%   0.0%   0.0%   0.0%   0.0%
       8    0.09   1.2%   0.2%  10.9%  87.7%   0.0%   0.0%   0.0%   0.0%
      10    0.12   3.0%   0.0%  11.1%  85.9%   0.0%   0.0%   0.0%   0.0%
      12    0.08   3.0%   0.2%   6.6%  90.3%   0.0%   0.0%   0.0%   0.0%
      14    0.09   4.2%   1.2%   0.8%  93.8%   0.0%   0.0%   0.0%   0.0%
      ---   ----  -----  -----  -----  -----  -----  -----  -----  -----
      avg   0.08   3.0%   1.0%   3.8%  92.2%   0.0%   0.0%   0.0%   0.0%
      
      System Page Size: 4Kbytes
      Memory: 38663044K (38379156K) real, 149420048K (148978096K) virtual, 26349848K f
      ree  Page# 1/63
      
      CPU TTY  PID USERNAME PRI NI   SIZE    RES STATE    TIME %WCPU  %CPU COMMAND
       4   ?  3926 root     152 20   213M 70040K run    694:47 10.05 10.04 cimprovagt
       4   ?  6855 user1  152 20  7704M  1125M run      4:36  9.31  9.29 java
       0   ?  6126 user2  152 20  2790M  1863M run     22:57  4.16  4.15 java
      Here is the Memory on the box

      Memory: 98132 MB (95.83 GB)


      Thanks
        • 1. Re: Oracle BPM 11.1.1.5 Performance issues
          899771
          After changing JVM settigns for soa cluster it's a bit better but still slow so wondering what other tweaks can be done on the JVM side. Here is what is for the SOA Cluster

          USER_MEM_ARGS="-server -Xms12928m -Xmx12928m -XX:PermSize=3072m -Xmn3232m -XX:+SXTElimination -XX:+UseParallelGC -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:-TraceClassLoading -XX:-TraceClassUnloading"


          We are running multiple instances on the same boxes and the total RAM on the machine is 95 GB on each box which is being shared across 4 cluster environments. For now other 3 cluster environments have the SOA Cluster JVM setting as

          USER_MEM_ARGS="-server -Xms4096m -Xmx4096m -XX:PermSize=1024m -Xmn1152m -XX:+SXTElimination -XX:+UseParallelGC -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:-TraceClassLoading -XX:-TraceClassUnloading"


          Any help on what else I can tweak or set in JVM to get a better performance.


          Thanks