1 Reply Latest reply on Aug 24, 2013 8:14 PM by jimE

    Web Application services failing to start (Foundation SS, Hyperion Admin)

    Benjamin Finkel
      Hello,

      Hyperion Essbase 11.1.1.3

      In my installation recently my web applications are failing to start properly. I'm following the start order and allowing services time to come up, but they are still reporting an error.

      1. OpenLDAP starts fine, and quickly.
      2. Hyperion Foundation Shared Services - Web Application will not start. It fails each time (as does Hyperion Administration Services - Web Application) with the same error message in the logs (see below).

      Nothing has changed on these servers in a long time. Any ideas?

      Thanks!

      startServer.log:

      ************ Start Display Current Environment ************
      Host Operating System is Windows Server 2003, version 5.2 build 3790 Service Pack 2
      Java version = J2RE 1.5.0 IBM J9 2.3 Windows Server 2003 x86-32 j9vmwi3223ifx-20090225 (JIT enabled)
      J9VM - 20090224_30451_lHdSMr
      JIT - 20081112_1511ifx1_r8
      GC - 200811_07, Java Compiler = j9jit23, Java VM name = IBM J9 VM
      was.install.root = D:\IBM\WebSphere\AppServer
      user.install.root = D:\Web\HEA\deployments\WebSphere6/profile
      Java Home = D:\IBM\WebSphere\AppServer\java\jre
      ws.ext.dirs = D:\IBM\WebSphere\AppServer\java\lib;D:\IBM\WebSphere\AppServer\classes;D:\IBM\WebSphere\AppServer\lib;D:\IBM\WebSphere\AppServer\installedChannels;D:\IBM\WebSphere\AppServer\lib\ext;D:\IBM\WebSphere\AppServer\web\help;D:\IBM\WebSphere\AppServer\deploytool\itp\plugins\com.ibm.etools.ejbdeploy\runtime
      Classpath = D:\Web\HEA\common\config\9.5.0.0\lib\hit-config.jar;D:\Web\HEA\common\config\9.5.0.0\lib\hit-common.jar;D:\IBM\WebSphere\AppServer\profiles\AppSrv01\properties;D:\IBM\WebSphere\AppServer\properties;D:\IBM\WebSphere\AppServer\lib\startup.jar;D:\IBM\WebSphere\AppServer\lib\bootstrap.jar;D:\IBM\WebSphere\AppServer/lib/j2ee.jar;D:\IBM\WebSphere\AppServer/lib/lmproxy.jar;D:\IBM\WebSphere\AppServer/lib/urlprotocols.jar;D:\IBM\WebSphere\AppServer\java\lib\tools.jar;D:\Web\HEA\products\Foundation\server\conf;D:\Web\HEA\deployments\WebSphere6\SharedServices9\config;D:\Web\HEA\common\JakartaCommons\commons-pool-1.3.jar;D:\Web\HEA\common\JDBC\DataDirect\3.7\lib\hyjdbc.jar;D:\Web\HEA\common\JakartaCommons\commons-dbcp-1.2.1.jar;D:\Web\HEA\deployments\WebSphere6\SharedServices9\config/lib;D:\Web\HEA\common\JDBC\DataDirect\3.7\lib;D:\Web\HEA\common\SAP\lib;D:\Web\HEA\common\config\9.5.0.0\lib\hit-config.jar;D:\Web\HEA\common\config\9.5.0.0\lib\hit-common.jar;
      Java Library path = D:\IBM\WebSphere\AppServer\java\jre\bin;.;D:\IBM\WebSphere\AppServer\java\jre\bin;D:\Web\HEA\common\SAP\bin;D:\Web\HEA\common\CSS\9.5.0.0\bin;D:\Web\HEA\common\CSS\9.5.0.0\bin;D:\Web\HEA\products\Essbase\EssbaseClient\bin;D:\Web\HEA\products\Essbase\EssbaseClient;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Web\HEA\common\ODBC-64\Merant\5.2\Drivers;D:\Web\HEA\products\Essbase\eas\console\bin;D:\Web\HEA\products\Essbase\eas\server\bin;C:\Program Files (x86)\Windows Imaging\;d:\PROGRA~1\BMCSOF~1\Patrol3\bin;D:\IBM\WebSphere\AppServer\bin;D:\IBM\WebSphere\AppServer\java\bin;D:\IBM\WebSphere\AppServer\java\jre\bin;D:\Web\HEA\common\config\9.5.0.0;D:\Web\HEA\products\Essbase\EssbaseClient\bin;D:\Web\HEA\products\Essbase\EssbaseClient;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Web\HEA\common\ODBC-64\Merant\5.2\Drivers;D:\Web\HEA\products\Essbase\eas\console\bin;D:\Web\HEA\products\Essbase\eas\server\bin;D:\Web\HEA\common\CSS\9.5.0.0\bin;D:\Web\HEA\products\Essbase\EssbaseClient\bin;D:\Web\HEA\products\Essbase\EssbaseClient;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Web\HEA\common\ODBC-64\Merant\5.2\Drivers;D:\Web\HEA\products\Essbase\eas\console\bin;D:\Web\HEA\products\Essbase\eas\server\bin;C:\Program Files (x86)\Windows Imaging\;d:\PROGRA~1\BMCSOF~1\Patrol3\bin
      Current trace specification = *=info:com.ibm.*=all
      ************* End Display Current Environment *************
      [2/20/13 9:45:00:032 EST] 0000000a ManagerAdmin I TRAS0017I: The startup trace state is *=info:com.ibm.*=all.
      [2/20/13 9:45:00:048 EST] 0000000a WsServerContr 1 Executing executeUtilityOnWindows with args: D:\Web\HEA\deployments\WebSphere6\profile\config\ hyslCell hyslNode SharedServices9
      [2/20/13 9:45:00:048 EST] 0000000a WsServerContr 1 Getting WindowsService obj with: SharedServices9 and D:\Web\HEA\deployments\WebSphere6\profile\config\ hyslCell hyslNode SharedServices9
      [2/20/13 9:45:00:048 EST] 0000000a WindowsServic > getWindowsServiceNameForServer() Entry
      [2/20/13 9:45:00:048 EST] 0000000a WindowsServic 1 In executeCommand, args are: D:\IBM\WebSphere\AppServer\bin\WASService.exe -getServiceName SharedServices9 -profilePath D:\Web\HEA\deployments\WebSphere6\profile
      [2/20/13 9:45:00:048 EST] 0000000a WindowsServic 1 Runtime.exec() returned process java.lang.ProcessImpl@161a161a
      [2/20/13 9:45:00:079 EST] 0000000a WindowsServic 1 retCode from childProcess.waitFor() is 0
      [2/20/13 9:45:00:079 EST] 0000000a WindowsServic 3 exitCode = 0
      [2/20/13 9:45:01:079 EST] 0000000a WindowsServic 1 Trying to new up WindowsService using serverName=SharedServices9, winServiceName=null
      [2/20/13 9:45:01:079 EST] 0000000a ManagerAdmin I TRAS0018I: The trace state has changed. The new trace state is *=info.
      [2/20/13 9:45:01:142 EST] 0000000a AdminTool A ADMU0128I: Starting tool with the hyslProfile profile
      [2/20/13 9:45:01:142 EST] 0000000a AdminTool A ADMU3100I: Reading configuration for server: SharedServices9
      [2/20/13 9:45:04:939 EST] 0000000a AdminTool A ADMU3200I: Server launched. Waiting for initialization status.
      [2/20/13 9:45:12:361 EST] 0000000a AdminTool A ADMU3011E: Server launched but failed initialization. startServer.log, SystemOut.log(or job log in zOS) and other log files under D:\Web\HEA\deployments\WebSphere6/profile\logs\SharedServices9 should contain failure information.

      SystemErr.log

      ************ Start Display Current Environment ************
      Log file started at: [2/20/13 9:45:09:705 EST]
      ************* End Display Current Environment *************
      [2/20/13 9:45:12:330 EST] 0000000a SystemErr R com.ibm.ws.exception.ConfigurationError: org.xml.sax.SAXParseException: Premature end of file.
      [2/20/13 9:45:12:330 EST] 0000000a SystemErr R      at com.ibm.ws.ssl.core.SSLComponentImpl.loadSecurityXMLFromConfig(SSLComponentImpl.java:314)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.ws.ssl.core.SSLComponentImpl.initialize(SSLComponentImpl.java:161)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.ws.runtime.component.ContainerImpl.initializeComponent(ContainerImpl.java:1338)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.ws.runtime.component.ContainerImpl.initializeComponents(ContainerImpl.java:1136)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.ws.runtime.component.ServerImpl.initialize(ServerImpl.java:356)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:178)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:140)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:461)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at java.lang.reflect.Method.invoke(Method.java:618)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:183)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:90)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:72)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at org.eclipse.core.internal.runtime.PlatformActivator$1.run(PlatformActivator.java:78)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:92)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:68)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:400)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:177)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at java.lang.reflect.Method.invoke(Method.java:618)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at org.eclipse.core.launcher.Main.invokeFramework(Main.java:336)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at org.eclipse.core.launcher.Main.basicRun(Main.java:280)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at org.eclipse.core.launcher.Main.run(Main.java:977)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:329)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R      at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:92)
      [2/20/13 9:45:12:345 EST] 0000000a SystemErr R Caused by: org.xml.sax.SAXParseException: Premature end of file.


      SystemOut.log

      ************ Start Display Current Environment ************
      WebSphere Platform 6.1 [BASE 6.1.0.23 cf230910.10] running with process name hyslCell\hyslNode\SharedServices9 and process id 1528
      Detailed IFix information: No IFixes applied to this build
      Host Operating System is Windows Server 2003, version 5.2 build 3790 Service Pack 2
      Java version = 1.5.0, Java Compiler = j9jit23, Java VM name = IBM J9 VM
      was.install.root = D:\IBM\WebSphere\AppServer
      user.install.root = D:\Web\HEA\DEPLOY~1\WEBSPH~1\profile
      Java Home = D:\IBM\WebSphere\AppServer\java\jre
      ws.ext.dirs = D:\IBM\WebSphere\AppServer/java/lib;D:\Web\HEA\DEPLOY~1\WEBSPH~1\profile/classes;D:\IBM\WebSphere\AppServer/classes;D:\IBM\WebSphere\AppServer/lib;D:\IBM\WebSphere\AppServer/installedChannels;D:\IBM\WebSphere\AppServer/lib/ext;D:\IBM\WebSphere\AppServer/web/help;D:\IBM\WebSphere\AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
      Classpath = D:\Web\HEA\DEPLOY~1\WEBSPH~1\profile/properties;D:\IBM\WebSphere\AppServer/properties;D:\IBM\WebSphere\AppServer/lib/startup.jar;D:\IBM\WebSphere\AppServer/lib/bootstrap.jar;D:\IBM\WebSphere\AppServer/lib/j2ee.jar;D:\IBM\WebSphere\AppServer/lib/lmproxy.jar;D:\IBM\WebSphere\AppServer/lib/urlprotocols.jar;D:\IBM\WebSphere\AppServer/deploytool/itp/batchboot.jar;D:\IBM\WebSphere\AppServer/deploytool/itp/batch2.jar;D:\IBM\WebSphere\AppServer/java/lib/tools.jar;D:\Web\HEA\common\config\9.5.0.0\lib\hit-config.jar;D:\Web\HEA\common\config\9.5.0.0\lib\hit-common.jar;D:\Web\HEA\common\SAP\lib;D:\Web\HEA\deployments\WebSphere6\SharedServices9\config;D:\Web\HEA\common\JDBC\DataDirect\3.7\lib\hyjdbc.jar;D:\Web\HEA\common\JakartaCommons\commons-pool-1.3.jar;D:\Web\HEA\common\JakartaCommons\commons-dbcp-1.2.1.jar
      Java Library path = D:\IBM\WebSphere\AppServer\java\jre\bin;.;D:\IBM\WebSphere\AppServer\java\jre\bin;D:\Web\HEA\common\SAP\bin;D:\Web\HEA\common\CSS\9.5.0.0\bin;D:\Web\HEA\common\CSS\9.5.0.0\bin;D:\Web\HEA\products\Essbase\EssbaseClient\bin;D:\Web\HEA\products\Essbase\EssbaseClient;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Web\HEA\common\ODBC-64\Merant\5.2\Drivers;D:\Web\HEA\products\Essbase\eas\console\bin;D:\Web\HEA\products\Essbase\eas\server\bin;C:\Program Files (x86)\Windows Imaging\;d:\PROGRA~1\BMCSOF~1\Patrol3\bin;D:\IBM\WebSphere\AppServer\bin;D:\IBM\WebSphere\AppServer\java\bin;D:\IBM\WebSphere\AppServer\java\jre\bin;D:\Web\HEA\common\config\9.5.0.0;D:\Web\HEA\products\Essbase\EssbaseClient\bin;D:\Web\HEA\products\Essbase\EssbaseClient;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Web\HEA\common\ODBC-64\Merant\5.2\Drivers;D:\Web\HEA\products\Essbase\eas\console\bin;D:\Web\HEA\products\Essbase\eas\server\bin;D:\Web\HEA\common\CSS\9.5.0.0\bin;D:\Web\HEA\products\Essbase\EssbaseClient\bin;D:\Web\HEA\products\Essbase\EssbaseClient;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Web\HEA\common\ODBC-64\Merant\5.2\Drivers;D:\Web\HEA\products\Essbase\eas\console\bin;D:\Web\HEA\products\Essbase\eas\server\bin;C:\Program Files (x86)\Windows Imaging\;d:\PROGRA~1\BMCSOF~1\Patrol3\bin
      ************* End Display Current Environment *************
      [2/20/13 9:45:09:876 EST] 0000000a ManagerAdmin I TRAS0017I: The startup trace state is *=info.
      [2/20/13 9:45:10:220 EST] 0000000a ManagerAdmin I TRAS0111I: The message IDs that are in use are deprecated
      [2/20/13 9:45:10:361 EST] 0000000a AdminInitiali A ADMN0015I: The administration service is initialized.
      [2/20/13 9:45:11:470 EST] 0000000a PluginConfigS I PLGC0057I: The plug-in configuration service started successfully.
      [2/20/13 9:45:11:486 EST] 0000000a ManagedObject I Processing extension point com.ibm.wsspi.extension.managed-object-metadata-collector
      [2/20/13 9:45:11:595 EST] 0000000a WorkSpaceMana A WKSP0500I: Workspace configuration consistency check is false.
      [2/20/13 9:45:11:736 EST] 0000000a SSLComponentI I CWPKI0001I: SSL service is initializing the configuration
      [2/20/13 9:45:12:205 EST] 0000000a ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl initialize FFDC0009I: FFDC opened incident stream file D:\Web\HEA\DEPLOY~1\WEBSPH~1\profile\logs\ffdc\SharedServices9_0000000a_13.02.20_09.45.12_0.txt
      [2/20/13 9:45:12:267 EST] 0000000a ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file D:\Web\HEA\DEPLOY~1\WEBSPH~1\profile\logs\ffdc\SharedServices9_0000000a_13.02.20_09.45.12_0.txt
      [2/20/13 9:45:12:267 EST] 0000000a SSLComponentI E CWPKI0011E: Failed to load security.xml resource from cell. The exception is org.xml.sax.SAXParseException: Premature end of file.