Forum Stats

  • 3,836,963 Users
  • 2,262,207 Discussions
  • 7,900,157 Comments

Discussions

On a clustered HFM environment, how to make hfm response to the correct web server where the request

Taco
Taco Member Posts: 49 Blue Ribbon
edited Oct 1, 2019 5:03AM in Financial Consolidation

Hi experts,

We're running a clustered environment on 11.1.2.4.

We have a load-balancer on the front, dispatch connections to 2 different web server, let's say web server A and B.

On both server A and B, also runs HFM server and have a clustered HFM web application.

We found that sometimes a user would get a blank screen when opening HFM applications,

and after some try & error process we found out it's because a session from the same user account already have a session connected to HFM via another web server.

For example if the user have connected to HFM application via web server A (session 1), and if he/she created another session and connected to HFM application from web server B (session 2), session 2 would fail to open HFM application.

I assume it's because the response from HFM application goes to web server A no matter the second connection from same user account comes from web server B.

The problem is how to change this behavior to make HFM response to the correct web server where the request came from?

Many thanks if someone can give some advice.

Eric

--20190930 update--

I was wrong, there's nothing to do with which web server I log into Workspace from, it's just a stuck thread issue that caused my Server B's HFM web app to stuck and later crashed.

The stuck thread in the end result in OOM error on server B, but before that the web server would still redirect request ot server B so the application hangs.

After server B's service is actually down, all traffic goes to server A and everything is fine again.

But still I need to solve the stuck thread issue, for now, I still don't understand what caused the stuck. I tried to increase the numbers of connection pool on weblogic server, and wait and see.

Eric

Message was edited by: Taco I'm here to update. I was wrong, there's nothing to do with

Tagged:

Answers

  • CBarbieri
    CBarbieri Member Posts: 1,011 Gold Trophy
    edited Sep 20, 2019 10:12AM

    If I read your question correctly, you assume there is a direct relationship between the web server clustering and the HFM application server clustering? There is no such relationship - even if the application server is installed and running on the same server as the web server. When a user logs into an HFM application, regardless of how they get to login (Workspace, Smart View, Financial Reports, or a custom tool), they will be assigned a session randomly on one of the application servers in the cluster. If the same user is already logged in to an application server, a feature in HFM called "sticky server" will override that random assignment and redirect their new login to the same HFM application server on which they are already connected.

    Separately, many companies use a web balancer to spread the workload across the web tier. So it is possible that the user can have a web session on server A but an application session on server B. Once you have an HFM application server cluster, which you must have once you have more than one application server, you have little control over which application server(s) the users will be assigned to. Individual users can override the cluster random assignment mechanism by going into their application preferences, but you can't control that centrally.

    If I have misunderstood your question please clarify and we can help you further. The occurrence of a blank screen is a separate issue from this topic of load balancing web and application server.

    - Chris

  • Taco
    Taco Member Posts: 49 Blue Ribbon
    edited Sep 20, 2019 10:54PM

    Hi Chris,

    Many thanks for reply. It's a little bit difficult to describe my question, so let me try to explain better.

    So, we have such environment:

    2 application server A and B, both have HFM running and clustered. Also, both have OHS installed so we can access workspace from both server. On the top, a load-balancer would dispatch the session, like you said, randomly to the web server.

    So the users always connect to the load-balancer via a given URL like http://abc.comnpany.com/workspace.

    Due to some report back from users, occasionally they would get a blank screen and unable to close it when entering a HFM application from workspace, we've performed this test:

    We have created 2 test account, say test1 and test2.

    We have user test1 log on workspace directly using web server A through a url like http://serverA:19000/workspace, and when opening HFM applications, everything is OK.

    Then we have user test1 log on workspace directly using web server B through a url like http://serverB:19000/workspace, without log out the first session, and when opening HFM applications, get a blank screen like I previously mentioned.

    Then we have user test2 log on workspace directly using web server B through a url like http://serverB:19000/workspace, and when opening HFM applications, everything is OK again, which means server B is actually working.

    Then we have user test2 log on workspace directly using web server A through a url like http://serverA:19000/workspace, and when opening HFM applications, blank screen.

    So I have this conclusion that when both server A and B working well, if I force user to create 2 sessions connecting via each web server, the later connection would fail to open applications from workspace.

    And I guess it's because the response from HFM web tier (weblogic) to OHS always goes to the first source of request. For user test1, it's web server A, and for user test2, it's web server B.

    I think there probably would have some chance for configuration, but I don't know how.

    I understand "Sticky server" configuration of HFM cluster, but like you said, it's controlling which application of a "specific session" always stick on to avoid inconsistency. But I'm facing another problem, I think.

    Again, many thanks for your reply, if you have any other suggestion, please reply me back!

    Eric 

  • Thanos A
    Thanos A Consolidation System Manager Member Posts: 1,443 Silver Trophy
    edited Sep 25, 2019 11:33AM

    Hi Eric,

    We had similar issues a few years ago.

    If I remember well, the issue was related with the number of Stuck Threads on the Weblogic. If the number of stuck threads was exceeding a specific number, the weblogic was loosing the OK status and we had to restart it.

    We permenantly fixed the issue by upgrading to the latest possible version of Weblogic.

    Cheers,

    Thanos

  • Taco
    Taco Member Posts: 49 Blue Ribbon
    edited Sep 26, 2019 12:12AM

    Hi Thanos,

    Thanks for reply.

    Can you recall which Weblogic version did you upgrade to?

    Eric

  • Thanos A
    Thanos A Consolidation System Manager Member Posts: 1,443 Silver Trophy
    edited Sep 26, 2019 5:23AM

    Hi,

    Currently we are at Weblogic 10.3 with the latest patch but I don't remember if it this or the previous patch that fixed the Weblogic issue.

    Hope it helps,

    Cheers,

    Thanos

  • Taco
    Taco Member Posts: 49 Blue Ribbon
    edited Sep 30, 2019 10:17AM

    Hi Thanos,

    We're also at 10.3.6 and also did the patch, still got the problem, not sure if it's because we're using MS sql server.

    We;ve increased the number of connection pool and will see if it helps.

    Eric

  • Thanos A
    Thanos A Consolidation System Manager Member Posts: 1,443 Silver Trophy
    edited Sep 30, 2019 11:38AM

    Hi,

    We did not think that it was an SQL server related issue.

    Have you checked your network? Are you in a hosted or a private environment?

    Thanos

  • Taco
    Taco Member Posts: 49 Blue Ribbon
    edited Oct 1, 2019 5:03AM

    Hi Thanos,

    It's a private environment.

    Which part of network do you think we should check?

    Pretty sure the connection between servers and databases are not blocked.

    Eric