Forum Stats

  • 3,726,642 Users
  • 2,245,234 Discussions
  • 7,852,335 Comments

Discussions

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Hyperion issues with multiple network cards - 11.1.2.4

tarijam
tarijam Member Posts: 8
edited May 2020 in EPM System Infrastructure

We have a new installation of 11.1.2.4.900 for Hyperion (HFM, Planning, Essbase, DRM, HFR and FDMEE).  It's been working great up till Go Live.  When the infrastructure team wants to add a 2nd NIC to the servers to be used for backups and monitoring Hyperion immediate stops working.  I believe the issue is that the Hyperion software/system is trying to talk on the 2nd NIC instead of the original.  The 2nd NIC does not allow for inter-server communication in this distributed environment.  We immediately set the binding order of the NICs to make sure the original card was the primary and the other was 2nd.  Restarted windows (2012 R2) and then started services back up but still that did not help.  We even tried re configuring all the "Deploy to Application Server" for all the items.  Still nothing. 

Does anyone have any suggestions?  This worked fine in our old PRod environment which was 11.1.2.4 installed on Windows 2008 R2. same NIC configuration.  As of now we have the 2nd NIC disabled in order for the apps to work BUT that means no backups!  We can't have that!

Thanks!

Tagged:
Madhusudhan. M

Answers

  • tarijam
    tarijam Member Posts: 8
    edited May 2020

    I may have resolved on my own.  Let's wait and see but so far the 2nd NIC is running and Hyperion is working fine.  I'll update the post when i confirm.

  • Srini+K
    Srini+K Member Posts: 253
    edited May 2020

    it looks as if you coded ip addresses in configuration. The registry already remembers the old ip addresses in configuration. had a similar issue where i need to block the ip addresses and manually assign them as a part of the setup with specific names assigned to the machines.

  • Vasantha Kumar
    Vasantha Kumar Member Posts: 39 Blue Ribbon
    edited May 2020

    In the primary NIC, go to IPV4 properties -> advanced -> uncheck "Automatic Metric" and enter '1' in the "Interface Metric". With this you should be able to use the secondary NIC as well.

    Madhusudhan. M
  • tarijam
    tarijam Member Posts: 8
    edited May 2020

    Thank you for your response.  I was not trying to change the IP Address or use a 2nd IP address.  The original NIC IP is good and i wanted to keep using that, but when i enabled a 2nd NIC, with different IP address (and ip range) it broke Hyperion.  The goal was for Hyperion to still use the original NIC and for the backup and monitoring utilities to use the 2nd NIC.  We set the binding order to the original first as that is the correct setup. 

  • tarijam
    tarijam Member Posts: 8
    edited May 2020

    so will putting a "1" on the primary NIC mean that it will be used first?  That is the goal.  I want Hyperion to only use the primary NIC.  Thanks.

  • Vasantha Kumar
    Vasantha Kumar Member Posts: 39 Blue Ribbon
    edited May 2020

    Yes. That is the purpose. I had come across the same exact scenario with one of my customer and after updating as I mentioned, it works fine. Please test and let me know if this is correct.

    Madhusudhan. M
  • Srini+K
    Srini+K Member Posts: 253
    edited May 2020

    looks like you have your preliminary answer from discussion above.  But one thing to point out. disable ip v6 addresses. There is a gain in performance i observed.

  • tarijam
    tarijam Member Posts: 8
    edited May 2020

    Thanks everyone.  Sorry i have been quiet for a bit.  So in the meantime we have set the binding order of the NIC cards where the original/primary is first, the 2nd NIC (newly added, causing the issue) is 2nd.  We set this order in the Configure Adapters screen using Advanced option.  We also set the Metric, as @Vasantha mentioned above to "1" on the primary cards.  We also updated the mod_ml_ohs.conf files on both FND servers and added the domain name (fqdn) for all the servers in our environment. Nothing is helping.  The servers will run fine with the 2nd NIC enabled, the issue is during service start (we restart our services every day as part of nightly maintenance).  Upon restart the services ALWAYS try to use the 2nd NIC CARD.  It's like they completely ignore the OS settings for binding and such (Windows 2012 R2).  I feel like there is some Hyperion setting that we are missing where we force Hyperion to only use that one NIC or particular IP address.

    The primary NIC is using 172.x.x.x and 2nd NIC uses 10.x.x.x.  I guess Hyperion favors the lower IP address?  Workaround right now is that we have to disable the 2nd NIC during that restart so only the primary NIC is active and then we have to go back and reenable the 2nd NIC again.  I've never seen this before and we had it working in our other 11.1.2.4 environment with 2 NICs before.  I appreciate any other thoughts or things to try.  Thanks so much!

  • Mudam
    Mudam Member Posts: 19
    edited May 2020

    Try to keep a higher value for the Interface Metric for the 2nd NIC , say 10 and see if it helps.

  • tarijam
    tarijam Member Posts: 8
    edited May 2020

    Unfortunately that didn't help.  Seems the problem is only on the foundation server.  When that 2nd NIC card is enabled DURING starting services it messes everything up! 

  • iArchSolutions-Joe
    iArchSolutions-Joe Member Posts: 133 Red Ribbon
    edited May 2020

    Good morning - is the issuing occurring for all services on the Foundation box or just the WebLogic services?  If only WebLogic service, you can set address binding inside of the WLS Admin server.  Please confirm which exact services on the server are having the issue. 

    Thanks!

    Joe Malewicki

    Certified Cloud Architect

    [email protected]

    203-880-4268

  • tarijam
    tarijam Member Posts: 8
    edited May 2020

    Hello Joe.  So the foundation server is also the WL admin server for the whole environment.  HFM, FDMEE, Planning, Essbase, DRM are on their own separate servers. I feel like the issue is when the WL admin service restarts AND the 2nd NIC is enabled, regardless of how we have configured that NIC (set binding order, set metric, etc.) it always uses the 2nd NIC to communicate and fails, as it can't reach the other servers.  So teh environment doesn't work. 

    I have already tried to set the "listening address" for all the other JVMs in WL Admin to see if that would help but it didn't seem to. 

    Thanks for your help!

  • iArchSolutions-Joe
    iArchSolutions-Joe Member Posts: 133 Red Ribbon
    edited May 2020

    OK so you did try to set the first NIC IP 172.xx as the listening address for each JVM hosted on that box?  As for troubleshooting this, you should try to focus on just one at first, save some time.  I always focus on the FoundationServices0 managed sever first then apply whatever fixes it afterward.  Is there anything in the foundationservices.log itself when the services crash/don't startup that you could share?  You'll see a ton of warnings about IP's addresses, etc... but if truly a WebLogic issue we should find something out there.  I hate to enable WLS debugging yet (nightmare of entries will get created) but was the standard log itself useful?  If you want to shoot it over we can look at it.  Either post non private snippets or if you feel more comfortable you can email it to me direct and we can take a look.

    Joe Malewicki

    Certified Cloud Architect

    [email protected]

    203-880-4268

  • iArchSolutions-Joe
    iArchSolutions-Joe Member Posts: 133 Red Ribbon
    edited May 2020

    One more silly and quick test thought.  Have you tried to as a test throw in a hosts file entry for the correct 172.xx IP's for the server names?  I.e. ALWAYS force the traffic to the 172.xx IP's for Hyperion server names?

  • tarijam
    tarijam Member Posts: 8
    edited May 2020

    yeah, these servers are only using hosts files.  They are not in a domain.  so Hosts files are required.  we used 172.x.x.x HOST FQDN setup. 

Sign In or Register to comment.