we faced one interesting situation with our application, running on clustered BEA Weblogic 9.2. We implemented some Singletons (ie. Singleton Service in a Weblogic cluster) and they are running fine. Most of the time.
Common scenarios: cluster is started -> the singletons run on one server; the server is brought down -> the singletons are started on the second server.
However, we had recently two situations where the Singletons started on both servers! Here they are:
1) We use Database Migration Basis and we have a dedicated table for Weblogic to capture current state of the singletons. Due to the fact that database (Oracle RAC) was partially available one of the Managed Server could not get the lease and probably assumed that no singleton is running, therefore started the singletons locally.
2) Network outage: we had several minutes where none of our servers could communicate between them (neither between SASU MS nor with the database). Consequence: each MS decided to start the singleton locally (after some time, which I assume is the waiting time). Once the network came back the servers could communicate again but the duplicate singletons are not deactivate, so they are running in parallel.
Anyone experienced the same or similar issues? How to prevent the above? Is the Consensus Migration Basis a better choice? (I think yes for #1 but not for situation #2).