We raised a SR over 3 weeks ago and have been trying to identify why the EPMServer0 crashes every 3 to 4 days and we need to restart it.
The memory is set as -Xms3072m –Xmx8160m
Monitoring the server it generally stays with the min level. We never say an OutOfMemory errors in the logs.
Also when it crashes I noticed that is Middleware folder size increased by 5Gig (too much wasted space). I noticed there are 246 folders with different dates under ./servers/EPMServer0/adr/diag/ofm/EPMSystem/EPMServer0/incident. The readme.txt had:
Incident detected using watch rule "UncheckedException":
Watch time: Dec 3, 2012 2:16:24 PM EST
Watch ServerName: EPMServer0
Watch RuleType: Log
Watch Rule: (SEVERITY = 'Error') AND ((MSGID = 'WL-101020') OR (MSGID = 'WL-101017') OR (MSGID = 'WL-000802') OR (MSGID = 'BEA-101020') O
R (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))
Watch DomainName: EPMSystem
DATE : Dec 3, 2012 2:16:24 PM EST
SERVER : EPMServer0
MESSAGE : [ServletContext@880365594[app:EAS module:easconsole path:/easconsole spec-version:2.5 version:18.104.22.168]] Servlet failed with Exception
java.lang.IllegalStateException: Response already committed
Any suggestions on identifying the crash and recommendations to clean the extra folders created.
You should be able to delete the incident dump files though I thought they were purged when they reached a certain size, how big is the incident directory?
It makes it more of a problem trying to find the issue as it looks like you have a compact deployment, if one of the web apps crash then it takes everything out, are all the incidents relating to EAS, is there a reason why you picked a compact deployment?
The incident total size is 562MB. Couple were created today, but the EPMServer0 is still running.
Will check the logs under EPMServer0 and the AdminServer domain level and delete older ones.
There are also many *.gz files under user_projects/domains/EPMSystem/servers/EPMServer0/logs/metrics but they total only 80MB (will delete older ones)
We installed in compact mode for development purpose. But for the next environment we will separate the components. (Seems this separation is highly recommended. Thks)
It is been hard to tell whether planning or Essbase is causing the issue, but believe it is Planning. When data is being entered in Planning forms is when it is the slowest.
FYI: we did enlarge the JDBC connection pools and database processes (1000) and open_cursors (1000) which fixed many issues and don't have DB connections issues.
We also modified planning JDBC connections:
JDBC_MIN_CONNECTIONS = 10 (from 1)
JDBC_MAX_CONNECTIONS = 45 (from 10 )
If you are experiencing issues then it may be worth moving from a compact deployment and deploy the web applications to their own managed server.
There is no point in deleting files which will be rotated, large dump files that are causing space problems can be deleted.
FYI: the version we have is 22.214.171.124 with weblogic server 10.3.6.0 and OHS.
I agree that deploying to separate managed servers is recommended, and mainly focus on Planning application initially.
see two options:
1. Since Planning is already installed and configured, the idea is to rerun the configuration wizard and reconfigure Planning on separate managed server and use the new default port. Will the config wizard remove and reinstall the Planning app correctly?
Note we currently have Planning applications already developed which will backup first.
Are there additional configurations to allow the links in Workspace to point to the new port?
I might need to modify mod_wl_ohs.conf to repoint <LocationMatch ^/HyperionPlanning>
2. We also have a separate machine running windows where EPMA and FDM are running using IIS. Currently no weblogic server is installed. I see we can install planning and configure it here, but then need to un-configure it from the Unix server and repoint OHS. Not sure how the unconfigure can be done and whether is option will help.
You can certainly deploy planning to its own managed server though the issue is that the compact deployment will still contain the planning web application which have to be removed using the WebLogic admin console, it is documented here - http://docs.oracle.com/cd/E17236_01/epm.1112/epm_install_1112200/frameset.htm?ch09s05.html
The OHS configuration should be updated if you run the configure web server again once planning has been deployed.