3 Replies Latest reply: May 28, 2013 9:23 AM by karthiksingh_dba RSS

    Every week Production is Non-Functional - How to resolve this?

    karthiksingh_dba
      Dear Legends,

      Our ENV:
      OS - Redhat Linux 64 bit
      DB - 11.2.0.3 RAC
      Weblogic - 10.3.6
      SOA Suite - 11.1.1.6 - we have 2 managed soa_servers and 1 admin server
      JVM - Sun JDK 1.6.0_29
      RAM - 24G

      For the past month we are facing the following issues very often
      1. EM is extremely slow - due to low space in DB - Tot - 47G Usd - 42G Free - 5G
      2. If we click any of the process sometimes it will come with a FABRIC WARNING Status with duration of 30mins - Is it due to the Dehydration space? We do purge every week but still we have more than 50000 records which is older than 90days.
      3. Often now a days we are getting stuck threads and utilizing the JVM fully and getting the servers to react badly. - Is there a way to UNSTUCK a STUCK THREAD MANUALLY?

      Please help me to avoid servers going down until we have our maintenance.

      Thanks,
      Karthik
        • 1. Re: Every week Production is Non-Functional - How to resolve this?
          gabelh
          Man,
          To solve both first and second issues you will have to implement database purge on your environment.
          Check this http://docs.oracle.com/cd/E25054_01/admin.1111/e10226/soaadmin_partition.htm
          After that you'll realize how important it is and how performance increases when you have a purge procedure implemented.

          To solve third issue, you'll have to take some thread dumps in the time you have a Stuck Thread on your JVM. If you post it here, I can help you analize it and detect what its causing this behavior.

          And NO, there is no way to unstuck a thread. The only way to solve it is to setup work manager to restart this jvm when a thread become stucked. But the right way its to investigate the thread issues.

          Kinds,
          Gabriel Abelha

          Edited by: gabelh on May 25, 2013 8:33 PM
          • 2. Re: Every week Production is Non-Functional - How to resolve this?
            karthiksingh_dba
            Hi,

            Thanks, I expected that purge would be the issue but the purge scripts are looking only for the states which are in completed state. It is not looking for RUNNING and FAULTED, FAULTED, FAULT and COMPLETE, FAULT and RUNNING.
            To solve third issue, you'll have to take some thread dumps in the time you have a Stuck Thread on your JVM. If you post it here, I can help you analize it and detect what its causing this behavior.
            Currently I have 2 Stuck Threads in soa_server1 and it is prevailing for 3days which shows WARNING Status. Thread dumps are here
            And NO, there is no way to unstuck a thread. The only way to solve it is to setup work manager to restart this jvm when a thread become stucked. But the right way its to investigate the thread issues.
            Restarting JVM wont affect other web service calls? What will be the servers reaction when a work manager is restarting the JVM?
            • 3. Re: Every week Production is Non-Functional - How to resolve this?
              karthiksingh_dba
              Hi,

              May I have an update on this. Because I need to know why the process is getting stuck and How to analyze the stuck process with TDA, TL, Samurai and Unstuck the thread or release some usage of JVM.

              Anyone Can reply to my question.

              Thanks,
              Karthik