Discussions
Categories
- 385.5K All Categories
- 5.1K Data
- 2.5K Big Data Appliance
- 2.5K Data Science
- 453.4K Databases
- 223.2K General Database Discussions
- 3.8K Java and JavaScript in the Database
- 47 Multilingual Engine
- 606 MySQL Community Space
- 486 NoSQL Database
- 7.9K Oracle Database Express Edition (XE)
- 3.2K ORDS, SODA & JSON in the Database
- 584 SQLcl
- 4K SQL Developer Data Modeler
- 188K SQL & PL/SQL
- 21.5K SQL Developer
- 45 Data Integration
- 45 GoldenGate
- 298.4K Development
- 4 Application Development
- 20 Developer Projects
- 166 Programming Languages
- 295K Development Tools
- 150 DevOps
- 3.1K QA/Testing
- 646.7K Java
- 37 Java Learning Subscription
- 37.1K Database Connectivity
- 201 Java Community Process
- 108 Java 25
- 22.2K Java APIs
- 138.3K Java Development Tools
- 165.4K Java EE (Java Enterprise Edition)
- 22 Java Essentials
- 176 Java 8 Questions
- 86K Java Programming
- 82 Java Puzzle Ball
- 65.1K New To Java
- 1.7K Training / Learning / Certification
- 13.8K Java HotSpot Virtual Machine
- 94.3K Java SE
- 13.8K Java Security
- 208 Java User Groups
- 25 JavaScript - Nashorn
- Programs
- 666 LiveLabs
- 41 Workshops
- 10.3K Software
- 6.7K Berkeley DB Family
- 3.6K JHeadstart
- 6K Other Languages
- 2.3K Chinese
- 207 Deutsche Oracle Community
- 1.1K Español
- 1.9K Japanese
- 474 Portuguese
Master node in 2-node BDB JE HA cluster unexpectedly shutting down feeder for replica with : " Reaso

Hi,
Mater node failed with this following :
2019-03-21 13:48:06.603 UTC INFO [broker-logic1-10.240.128.66] Shutting down feeder for replica broker-logic2-10.240.136.211 Reason: Expected bytes: 6 read bytes: 0 write time: 668ms Avg write time: 5us
2019-03-21 13:48:06.604 UTC INFO [broker-logic1-10.240.128.66] Feeder Output for broker-logic2-10.240.136.211 soft shutdown initiated.
2019-03-21 13:48:10.604 UTC WARNING [broker-logic1-10.240.128.66] Soft shutdown failed for thread:Thread[Feeder Output for broker-logic2-10.240.136.211,5,main] after waiting for 4000ms resorting to interrupt.
2019-03-21 13:48:18.604 UTC SEVERE [broker-logic1-10.240.128.66] Thread[Feeder Output for broker-logic2-10.240.136.211,5,main] shutdown via interrupt FAILED. Thread still alive despite waiting for 8000ms.
...
2019-03-21 13:48:19.047 UTC INFO [broker-logic1-10.240.128.66] Releasing commit block latch
2019-03-21 13:48:19.075 UTC SEVERE [broker-logic1-10.240.128.66]
com.sleepycat.je.EnvironmentWedgedException: (JE 7.5.11) Environment must be closed, caused by: com.sleepycat.je.EnvironmentWedgedException: Environment invalid because of previous exception: (JE 7.5.11) broker-logic1-10.240.128.66(1):/stage/broker-logic1 Thread[Feeder Output for broker-logic2-10.240.136.211,5,main] shutdown via interrupt FAILED. Thread still alive despite waiting for 8000ms. WEDGED: An internal thread could not be stopped. The current process must be shut down and restarted before re-opening the Environment. A full thread dump has been logged. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentWedgedException.wrapSelf(EnvironmentWedgedException.java:77)
at com.sleepycat.je.EnvironmentWedgedException.wrapSelf(EnvironmentWedgedException.java:49)
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1750)
at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:475)
at com.sleepycat.je.log.LogManager.logItem(LogManager.java:425)
at com.sleepycat.je.log.LogManager.log(LogManager.java:340)
at com.sleepycat.je.tree.LN.logInternal(LN.java:716)
at com.sleepycat.je.tree.LN.log(LN.java:448)
at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1448)
at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:1170)
at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:995)
at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:489)
at com.sleepycat.je.cleaner.Cleaner.doClean(Cleaner.java:670)
at com.sleepycat.je.dbi.EnvironmentImpl.invokeCleaner(EnvironmentImpl.java:2299)
at com.sleepycat.je.Environment.cleanLogFile(Environment.java:1839)
at com.tencent.hippo.broker.service.impl.DefStoreManagerService$CleanLogFileTask.run(DefStoreManagerService.java:956)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.sleepycat.je.EnvironmentWedgedException: Environment invalid because of previous exception: (JE 7.5.11) broker-logic1-10.240.128.66(1):/stage/broker-logic1 Thread[Feeder Output for broker-logic2-10.240.136.211,5,main] shutdown via interrupt FAILED. Thread still alive despite waiting for 8000ms. WEDGED: An internal thread could not be stopped. The current process must be shut down and restarted before re-opening the Environment. A full thread dump has been logged. Environment is invalid and must be closed.
at com.sleepycat.je.utilint.StoppableThread.shutdownThread(StoppableThread.java:386)
at com.sleepycat.je.rep.impl.node.Feeder.shutdown(Feeder.java:570)
at com.sleepycat.je.rep.impl.node.Feeder$InputThread.run(Feeder.java:762)
It's looks like there are two problem:
1. master node broker-logic1-10.240.128.66 want to read 6 bytes but read 0 bytes actually
2. the feeder want to shutdown but failed.
What could cause "Expected bytes: 6 read bytes: 0" , is there any way to solve this problem ?
Best Regards,
Mayo
Answers
-
> 2019-03-21 13:48:06.603 UTC INFO [broker-logic1-10.240.128.66] Shutting down feeder for replica broker-logic2-10.240.136.211 Reason: Expected bytes: 6 read bytes: 0 write time: 668ms Avg write time: 5us
This means the replica connection was closed, it is not a error for this node. Check the replica's log.
> Caused by: com.sleepycat.je.EnvironmentWedgedException: Environment invalid because of previous exception: (JE 7.5.11) broker-logic1-10.240.128.66(1):/stage/broker-logic1 Thread[Feeder Output for broker-logic2-10.240.136.211,5,main] shutdown via interrupt FAILED. Thread still alive despite waiting for 8000ms. WEDGED: An internal thread could not be stopped. The current process must be shut down and restarted before re-opening the Environment. A full thread dump has been logged. Environment is invalid and must be closed.
As the message mentions, a full thread dump was logged that may provide more insight.
--mark