1 Reply Latest reply on Sep 28, 2012 3:22 PM by Kalyan Pasupuleti-Oracle

    WLS failed - quick question

    901647
      Hi All

      We see these errors.
      What do they mean ?
      What are used in the .DAT files ?
      Why are they failing ?

      thanks

      <Sep 27, 2012 7:41:15 AM PDT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
      <Sep 27, 2012 7:41:15 AM PDT> <Emergency> <Store> <BEA-280060> <The persistent store "_WLS_AdminServer" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file WLSADMINSERVER000000.DAT.
      weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file WLSADMINSERVER000000.DAT.
      at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
      at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
      at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
      at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)
      at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191)
      Truncated. see log file for complete stacktrace
      Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
      at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
      at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54)
      at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
      at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
      at weblogic.store.io.file.StoreFile.openDirect(StoreFile.java:168)
      Truncated. see log file for complete stacktrace
      >
      No matching store mbean found for =_WLS_AdminServer, must be a default store.
      <Sep 27, 2012 7:41:15 AM PDT> <Critical> <Health> <BEA-310006> <Critical Subsystem PersistentStore._WLS_AdminServer has failed. Setting server state to FAILED.
      Reason: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file WLSADMINSERVER000000.DAT.>
      <Sep 27, 2012 7:41:15 AM PDT> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: health of critical service 'PersistentStore._WLS_AdminServer' failed>
      <Sep 27, 2012 7:41:15 AM PDT> <Error> <Store> <BEA-280061> <The persistent store "_WLS_AdminServer" could not be deployed: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file WLSADMINSERVER000000.DAT.
      weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file WLSADMINSERVER000000.DAT.
      at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
      at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
      at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
      at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)
      at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191)
      Truncated. see log file for complete stacktrace
      Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
      at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
      at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54)
      at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
      at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
      at weblogic.store.io.file.StoreFile.openDirect(StoreFile.java:168)
      Truncated. see log file for complete stacktrace
      >
      <Sep 27, 2012 7:41:15 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
      <Sep 27, 2012 7:41:15 AM PDT> <Error> <> <BEA-000000> <THREAD DUMP from JVM taken at 'Thu Sep 27 07:41:15 2012'


      2012-09-27 07:41:15
      Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):

      "Timer-2" daemon prio=10 tid=0x00007f2b5c33e800 nid=0x19fd in Object.wait() [0x00007f2b8135d000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00007f2bf8da0400> (a java.util.TaskQueue)
      at java.lang.Object.wait(Object.java:502)
      at java.util.TimerThread.mainLoop(Timer.java:505)
      - locked <0x00007f2bf8da0400> (a java.util.TaskQueue)
      at java.util.TimerThread.run(Timer.java:484)

      "Runtime Authorization Event Notifier" daemon prio=10 tid=0x00007f2b5c559000 nid=0x19fc in Object.wait() [0x00007f2b8145e000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00007f2bf8da0940> (a oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl)
      at java.lang.Object.wait(Object.java:502)
      at oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl.run(RuntimeEventNotifierImpl.java:230)
      - locked <0x00007f2bf8da0940> (a oracle.security.jps.az.internal.runtime.notifier.RuntimeEventNotifierImpl)
      at java.lang.Thread.run(Thread.java:636)

      "JMX Framework document pooling thread" daemon prio=10 tid=0x00007f2b7c01c800 nid=0x19fb waiting on condition [0x00007f2b8155f000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
      at java.lang.Thread.sleep(Native Method)
      at oracle.as.config.notification.filesystem.WatchingDocumentChangeNotifier.run(WatchingDocumentChangeNotifier.java:253)
      at java.lang.Thread.run(Thread.java:636)

      "JMX FRAMEWORK Domain Runtime MBeanServer pooling thread" daemon prio=10 tid=0x00007f2b5c45a000 nid=0x19f7 waiting on condition [0x00007f2b81660000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
      at java.lang.Thread.sleep(Native Method)
      at oracle.as.jmx.framework.wls.spi.JMXFrameworkProviderImpl$2.run(JMXFrameworkProviderImpl.java:1022)
      at java.lang.Thread.run(Thread.java:636)

      "JPS FileChangeNotifier<clean-only>" daemon prio=10 tid=0x00007f2b5c38e800 nid=0x19f6 waiting on condition [0x00007f2b81862000]
      java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00007f2bbd7d9ec8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2081)
      at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:688)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:681)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:636)

      "DoSManager" daemon prio=10 tid=0x00007f2b78390000 nid=0x19f5 waiting on condition [0x00007f2b81761000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
      at java.lang.Thread.sleep(Native Method)
      at com.octetstring.vde.DoSManager.run(DoSManager.java:433)

      "VDE Transaction Processor Thread" daemon prio=10 tid=0x00007f2b5c0f7800 nid=0x19f3 in Object.wait() [0x00007f2b81963000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00007f2bbd82bd30> (a com.octetstring.vde.backend.standard.TransactionProcessor)
      at java.lang.Object.wait(Object.java:502)
      at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367)
      - locked <0x00007f2bbd82bd30> (a com.octetstring.vde.backend.standard.TransactionProcessor)
      at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)

      "[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f2c007b8800 nid=0x19f1 waiting on condition [0x00007f2b81a64000]
      java.lang.Thread.State: RUNNABLE
      at weblogic.platform.SunVM.fileThreadDump0(Native Method)
      - locked <0x00007f2bbd47cc70> (a weblogic.platform.SunVM)
      at weblogic.platform.SunVM.threadDump(SunVM.java:102)
      at weblogic.platform.SunVM.threadDump(SunVM.java:135)
      at weblogic.t3.srvr.T3Srvr.logThreadDump(T3Srvr.java:299)
      at weblogic.t3.srvr.T3Srvr.failed(T3Srvr.java:226)
      - locked <0x00007f2bf8d9da10> (a weblogic.t3.srvr.T3Srvr)
      at weblogic.health.HealthMonitorService$2.run(HealthMonitorService.java:188)
      at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

      "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x00007f2b6004a800 nid=0x19f0 waiting for monitor entry [0x00007f2b81b65000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:95)
      - waiting to lock <0x00007f2bbd8573c8> (a weblogic.socket.PosixSocketMuxer$1)
      at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

      "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=0x00007f2b60013000 nid=0x19ef waiting for monitor entry [0x00007f2b81c66000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:95)
      - waiting to lock <0x00007f2bbd8573c8> (a weblogic.socket.PosixSocketMuxer$1)
      at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
      at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

      :
      "Timer-0" daemon prio=10 tid=0x00007f2c00b25000 nid=0x19df in Object.wait() [0x00007f2b9c9f0000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00007f2bf8d9e378> (a java.util.TaskQueue)
      at java.lang.Object.wait(Object.java:502)
      at java.util.TimerThread.mainLoop(Timer.java:505)
      - locked <0x00007f2bf8d9e378> (a java.util.TaskQueue)
      at java.util.TimerThread.run(Timer.java:484)

      "Low Memory Detector" daemon prio=10 tid=0x00007f2c0009d000 nid=0x19dd runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "CompilerThread1" daemon prio=10 tid=0x00007f2c0009a000 nid=0x19dc waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "CompilerThread0" daemon prio=10 tid=0x00007f2c00097800 nid=0x19db waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "Signal Dispatcher" daemon prio=10 tid=0x00007f2c00095800 nid=0x19da runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "Finalizer" daemon prio=10 tid=0x00007f2c00074800 nid=0x19d9 in Object.wait() [0x00007f2c051aa000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00007f2bbd87a9d0> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
      - locked <0x00007f2bbd87a9d0> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

      "Reference Handler" daemon prio=10 tid=0x00007f2c00072800 nid=0x19d8 in Object.wait() [0x00007f2c052ab000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00007f2bbdade268> (a java.lang.ref.Reference$Lock)
      at java.lang.Object.wait(Object.java:502)
      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
      - locked <0x00007f2bbdade268> (a java.lang.ref.Reference$Lock)

      "main" prio=10 tid=0x00007f2c00009800 nid=0x19d4 waiting for monitor entry [0x00007f2c06768000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at weblogic.t3.srvr.T3Srvr.setFailedState(T3Srvr.java:318)
      - waiting to lock <0x00007f2bf8d9da10> (a weblogic.t3.srvr.T3Srvr)
      at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:888)
      at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
      at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
      at weblogic.Server.main(Server.java:71)

      "VM Thread" prio=10 tid=0x00007f2c0006d800 nid=0x19d7 runnable

      "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2c00014000 nid=0x19d5 runnable

      "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2c00015800 nid=0x19d6 runnable

      "VM Periodic Task Thread" prio=10 tid=0x00007f2c0009f000 nid=0x19de waiting on condition

      >
      <Sep 27, 2012 7:41:15 AM PDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

      There are 1 nested errors:

      weblogic.management.DeploymentException:
      at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:129)
      at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191)
      at weblogic.store.admin.DefaultStoreService.start(DefaultStoreService.java:60)
      at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
      at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
      Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
      at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
      at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54)
      at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
      at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
      at weblogic.store.io.file.StoreFile.openDirect(StoreFile.java:168)
      at weblogic.store.io.file.Heap.openStoreFile(Heap.java:398)
      at weblogic.store.io.file.Heap.open(Heap.java:325)
      at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
      at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
      at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
      at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)
      ... 6 more

      >
      <Sep 27, 2012 7:41:15 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
      <Sep 27, 2012 7:41:15 AM PDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
      <Sep 27, 2012 7:41:15 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

      Edited by: 898644 on Sep 27, 2012 8:17 AM

      Edited by: 898644 on Sep 27, 2012 8:19 AM