Our application uses third party cache to store login related information after authentication. Earlier since it was a single jvm deployment and cache configured was stand alone it was working fine.
Cluster setup with Node1 and Node2 with Apache web server
1. User1 accesses the application, gets directed to Node1 and standalone cache store the information about successful authentication.
2. Node 1 goes down. Cache is lost. But other HTTP session data got replicated to Node2.
3. User1 accesses the application and gets redirected to login page(as Cache is not distributed, When Node1 goes down cache is lost and no authentication data on Node2 for user1).
After enabling the distributed cache, data stays as long as the cache server is alive. So authentication data is not lost. So now failover is happening.
Note: Now i could make out that from weblogic side, HTTP session replication was happening before as well (May be when started this thread as well). Its just the authentication data which was getting stored in cache that was the issue.