JRockit JVM Freezes on server start and WLST domain configuration
697311Apr 21 2009 — edited Jun 17 2009Hi,
Here is our configuration:
OS:
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.3 (Tikanga)
uname -a
Linux ********* 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
rpm -qa | grep glibc
glibc-2.5-34
glibc-common-2.5-34
glibc-2.5-34
We are using WLS 10.3 with 64 bit OS and 64 bit JRockit JVM on Xeon system.
The JVM freezes for about 10m when I am trying to start the server. I also see the same behavior when I am trying to create the domain using WLST. Here is the thread dump during the freeze. The JVM eventually recovers from this and continues processing. During domain configuration, the freeze happens when it is trying to write the domain files to disk. All writes are happening to local disk.
===== FULL THREAD DUMP ===============
Mon Apr 20 18:37:41 2009
BEA JRockit(R) R27.6.0-50_o-100423-1.6.0_05-20080626-2104-linux-x86_64
"Main Thread" id=1 idx=0x4 tid=30539 prio=5 alive, in native
at jrockit/io/FileNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BII)I(Native Method)
at jrockit/io/FileNativeIO.readBytes(FileNativeIO.java:46)
at java/io/FileInputStream.readBytes([BII)I(FileInputStream.java)
at java/io/FileInputStream.read(FileInputStream.java:199)
at sun/security/provider/NativePRNG$RandomIO.readFully(NativePRNG.java:185)
at sun/security/provider/NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:202)
^-- Holding lock: java/lang/Object@0x8577948[biased lock]
at sun/security/provider/NativePRNG$RandomIO.access$300(NativePRNG.java:108)
at sun/security/provider/NativePRNG.engineGenerateSeed(NativePRNG.java:102)
at java/security/SecureRandom.generateSeed(SecureRandom.java:495)
at com/bea/security/utils/random/AbstractRandomData.ensureInittedAndSeeded(AbstractRandomData.java:83)
^-- Holding lock: com/bea/security/utils/random/SecureRandomData@0x95f5db0[recursive]
at com/bea/security/utils/random/AbstractRandomData.getRandomBytes(AbstractRandomData.java:97)
^-- Holding lock: com/bea/security/utils/random/SecureRandomData@0x95f5db0[biased lock]
at com/bea/security/utils/random/AbstractRandomData.getRandomBytes(AbstractRandomData.java:92)
at weblogic/security/Salt.getRandomBytes(Salt.java:18)
at weblogic/security/internal/encryption/JSafeSecretKeyEncryptor.decryptSecretKey(JSafeSecretKeyEncryptor.java:108)
at weblogic/security/internal/encryption/JSafeEncryptionServiceImpl.<init>(JSafeEncryptionServiceImpl.java:206)
at weblogic/security/internal/encryption/JSafeEncryptionServiceFactory.getEncryptionService(JSafeEncryptionServiceFactory.java:23)
at weblogic/security/internal/SerializedSystemIni.getEncryptionService(SerializedSystemIni.java:214)
at weblogic/security/internal/SerializedSystemIni.getExistingEncryptionService(SerializedSystemIni.java:240)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java/lang/reflect/Method.invoke(Method.java:597)
at weblogic/descriptor/DescriptorManager$SecurityServiceImpl$SecurityProxy.<init>(DescriptorManager.java:149)
at weblogic/descriptor/DescriptorManager$SecurityServiceImpl$SecurityProxy.instance(DescriptorManager.java:137)
at weblogic/descriptor/DescriptorManager$SecurityServiceImpl.isEncrypted(DescriptorManager.java:108)
at weblogic/descriptor/internal/AbstractDescriptorBean._isEncrypted(AbstractDescriptorBean.java:968)
at weblogic/management/configuration/SecurityConfigurationMBeanImpl.setCredentialEncrypted(SecurityConfigurationMBeanImpl.java:1557)
at weblogic/management/configuration/SecurityConfigurationMBeanImpl.setCredentialEncryptedAsString(SecurityConfigurationMBeanImpl.java:781)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java/lang/reflect/Method.invoke(Method.java:597)
at com/bea/staxb/runtime/internal/util/ReflectionUtils.invokeMethod(ReflectionUtils.java:48)
at com/bea/staxb/runtime/internal/RuntimeBindingType$BeanRuntimeProperty.setValue(RuntimeBindingType.java:531)
at com/bea/staxb/runtime/internal/MultiIntermediary.setValue(MultiIntermediary.java:80)
at com/bea/staxb/runtime/internal/ByNameRuntimeBindingType$ElementQNameProperty.fill(ByNameRuntimeBindingType.java:348)
at com/bea/staxb/runtime/internal/LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:139)
at com/bea/staxb/runtime/internal/ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
at com/bea/staxb/runtime/internal/AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:47)
at com/bea/staxb/runtime/internal/LiteralUnmarshalResult.unmarshalElementProperty(LiteralUnmarshalResult.java:164)
at com/bea/staxb/runtime/internal/LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:136)
at com/bea/staxb/runtime/internal/ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
at com/bea/staxb/runtime/internal/AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:47)
at com/bea/staxb/runtime/internal/UnmarshalResult.unmarshalBindingType(UnmarshalResult.java:189)
at com/bea/staxb/runtime/internal/UnmarshalResult.unmarshalDocument(UnmarshalResult.java:159)
at com/bea/staxb/runtime/internal/UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:65)
at weblogic/descriptor/internal/MarshallerFactory$1.createDescriptor(MarshallerFactory.java:95)
at weblogic/descriptor/BasicDescriptorManager.createDescriptor(BasicDescriptorManager.java:240)
at weblogic/management/provider/internal/DescriptorManagerHelper.loadDescriptor(DescriptorManagerHelper.java:63)
at weblogic/management/provider/internal/RuntimeAccessImpl$IOHelperImpl.parseXML(RuntimeAccessImpl.java:683)
at weblogic/management/provider/internal/RuntimeAccessImpl.parseNewStyleConfig(RuntimeAccessImpl.java:268)
at weblogic/management/provider/internal/RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:115)
at weblogic/management/provider/internal/RuntimeAccessService.start(RuntimeAccessService.java:39)
at weblogic/t3/srvr/ServerServicesManager.startService(ServerServicesManager.java:459)
at weblogic/t3/srvr/ServerServicesManager.startInStandbyState(ServerServicesManager.java:164)
^-- Holding lock: java/lang/Class@0x939df30[biased lock]
at weblogic/t3/srvr/T3Srvr.initializeStandby(T3Srvr.java:711)
at weblogic/t3/srvr/T3Srvr.startup(T3Srvr.java:482)
at weblogic/t3/srvr/T3Srvr.run(T3Srvr.java:440)
at weblogic/Server.main(Server.java:67)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
<<Truncated for brevity>>
===== END OF THREAD DUMP ===============
The same process works fine on a VM with the same OS. Any ideas what might be going on?
Thanks,
Chengi