SOA em taking longer than usual to login due to stuck threads.

User470823-OC

Hello

When we are trying to SOA EM console it took longer than expected to login to the console we dont see any error at login but the ball just spins for ever , only thing I could see in the logs (AdminServer logs) is

####<May 30, 2014 3:03:25 PM PDT> <Error> <WebLogicServer> <tstrdsoaap132.tst.nintendo.com> <AdminServer> <[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuni

ng)'> <<WLS Kernel>> <> <3b367adad71a6c0c:-1583500c:14644b78c4b:-8000-00000000000015f9> <1401487405631> <BEA-000337> <[STUCK] ExecuteThread: '23' for queue: 'weblogic.kernel.Default

(self-tuning)' has been busy for "641" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 641530 ms

[

POST /em/faces/targetauth/emasLogin?_adf.ctrl-state=fbxlke858_4&target=Farm_aosts02&type=oracle_ias_farm&ipMsgID=83558be9 HTTP/1.1

Connection: keep-alive

Content-Length: 557

Origin: https://tstrdsoaap132.tst.nintendo.com:7443

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36

Adf-Rich-Message: true

Content-Type: application/x-www-form-urlencoded; charset=UTF-8

Accept: */*

Referer: https://tstrdsoaap132.tst.nintendo.com:7443/em/faces/targetauth/emasLogin?target=Farm_aosts02&type=oracle_ias_farm&FromContextInitFilter=true&_afrLoop=2281455357092821&_afrW

indowMode=0&_afrWindowId=null

Accept-Encoding: gzip,deflate,sdch

Accept-Language: en-US,en;q=0.8

Cookie: JSESSIONID=n0TWTL9VH1T4p1b7j30hdNTl3t2DkGyYPg7g9RBzydhlhfLdvYgh!1282094521; NINTENDO=205.166.76.241.1399391558961511; fsr.r=%7B%22d%22%3A90%2C%22i%22%3A%22d445cf1-83295732-b4

bf-0262-2d538%22%2C%22e%22%3A1400196054867%7D; AEBPD01=5ExTzxUuN4RBEwhYVk3rL9vmaF; fsr.s=%7B%22v2%22%3A1%2C%22v1%22%3A1%2C%22rid%22%3A%22d445cf1-83295732-b4bf-0262-2d538%22%2C%22ru%2

2%3A%22http%3A%2F%2Fwww.nintendo.com%2Fconsumer%2Fassets%2Fregion_select.jsp%22%2C%22r%22%3A%22www.nintendo.com%22%2C%22st%22%3A%22%22%2C%22to%22%3A5%2C%22mid%22%3A%22d445cf1-8329611

5-927a-8c46-41b67%22%2C%22rt%22%3Afalse%2C%22rc%22%3Atrue%2C%22c%22%3A%22http%3A%2F%2Fwww.nintendo.com%2Fconsumer%2Findex.jsp%22%2C%22pv%22%3A18%2C%22lc%22%3A%7B%22d0%22%3A%7B%22v%22

%3A18%2C%22s%22%3Atrue%7D%7D%2C%22cd%22%3A0%2C%22f%22%3A1399924791343%2C%22sd%22%3A0%2C%22l%22%3A%22en%22%2C%22i%22%3A-1%7D; aebts02=pcAZXYBDwauF8vbdbD3f3V40aX; s_vi=[CS]v1|29B5D9DE0

5011DC1-4000010F2005441A[CE]; atgPlatoStop=1; aebdv03=fLQ2RUjzEqfEoCRwaD2NuFzMeL; s_cc=true; s_sq=ncomstore%2Cncomglobal%3D%2526pid%253Dhttps%25253A%25252F%25252Fstore.nintendo.com%2

5252Fng3%25252Fus%25252Fpo%25252Fbrowse%25252Fsubcategory.jsp%25253FcategoryId%25253Dcat10019%2526oid%253Dhttps%25253A%25252F%25252Fstore.nintendo.com%25252Fng3%25252Fus%25252Fpo%25252Fbrowse%25252FproductDetailColorSizePicker.jsp%25253FcategoryNav%25253Dtrue%252526navAct%2526ot%253DA

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:

null>

What could be the reason for this stuck threads , I had to restart AdminServer to get the stuck threads cleared.

Albin

May be the below post will help you to resolve the issue

Albin's Blog: Increasing the performance of EM console in Oracle SOA 11g - Part1

Albin's Blog: Increasing the performance of EM console in Oracle SOA 11g - Part2

Regards

Albin I

http://www.albinsblog.com

Mark the question accordingly "Answered" or "Helpful"

User470823-OC

Thanks I will try these options , and get back

icharlto -Oracle FMw SOA-Oracle

Hello

Slow EM performance is usually down to a couple of things;

1. instances have not been purged, resulting in slowness

2. metric information is still set to poll with default values, thus slowing down further.

For metrics;

1. http://docs.oracle.com/cd/E15586_01/integration.1111/e10226/appx_trouble.htm#SOAAG37214

2. http://docs.oracle.com/cd/E15586_01/fusionapps.1111/e14496/soa_trouble.htm#BABFEAGD

For purging;

Ensure you're on a recent release, like Patchset 6 (11.1.1.7) and have all the mandatory patches related to purging

- see the Purging tab >> Oracle Fusion Middleware (FMW) SOA 11g Infrastructure Database: Installation, Maintenance and Administration Guide (Doc ID 1384379.1)

Thanks

iain

User470823-OC

Albin

Albin's Blog: Increasing the performance of EM console in Oracle SOA 11g - Part1

Disable fetching of instance and fault count metrics - we currently have it for 10 min less that what you suggested.

Enable fmwctl discovery cache:

I will have to discuss internally for this option

Albin's Blog: Increasing the performance of EM console in Oracle SOA 11g - Part2

<collectorConfiguration>

    <prefetch intervalSeconds="15" removeCycle="2" isDefault="true"/>

    <prefetch intervalSeconds="300" removeCycle="3"/>

    <discover intervalSeconds="180"/>

  </collectorConfiguration>

  <dumpConfiguration>

    <dump intervalSeconds="10800" maxSizeMBytes="75" enabled="true"/>

  </dumpConfiguration>

There are current settings

User470823-OC

icharlto -Oracle FMw SOA-Oracle

We are currently running on SOA 11.1.1.6 there is no plan to upgrade to 11.1.1.7 as we found some bugs in it.

icharlto -Oracle FMw SOA-Oracle

PS5 instead of PS6 isOK, as the same applies to your version.
Tune out the EM metrics and look into instances for purging that match your business retention requirements.

cheers

iain

User470823-OC

I have an SR opened on the same issue, the EM on some environments takes more than 10min to even login.

User470823-OC

And to answer your question on purging is scheduled to be done twice every week on these environments.

Raj__K

Though purging is executed, ensure whether all the instances related information are removed from soa database. Some version of soa suite has bugs related to purging.

Dhaval Parikh-Oracle

Hi ,

Can you please also investigate the stuck threads and see what is the root cause for the stuck threads? This can be done by taking 5-6 thread dumps at interval of 5 mins each to check what threads are showing up in all of them or most of them. this might help suggest appropriate solution.

Also please refer the Tuning guide and see if any applicable tuning is missing in your env.

Oracle&amp;reg; Fusion Middleware Performance and Tuning Guide 11g Release 1 (11.1.1.7.0) - Contents