7 Replies Latest reply: Sep 11, 2012 10:13 AM by 958766 RSS

    DR server still showing consolidation users and messages from production

    958766
      We are doing a DR exercise. My first. We copied the production database to the dr servers. Since they are on the same domain, we put entries in the host files on all our servers to spoof the prod server names to point to dr servers. Moved selected data files for essbase, etc to the dr application disks.

      Everything appears to be working except for HFM. I still see production users activity and messages in the shared services console when I go to administer, show consolidation user activity or messages.

      I have searched the registry, all database tables I could think of, and the application disks on the HFM and shared services services looking for a hardcoded IP and can not find one.

      The message below came out of the hfm config log which has the production database name (which when pinged points to the DR database serrver) but the IP listed is the production IP. Any suggestions?

      (Jul 30, 2010, 10:21:25 AM), com.hyperion.config.wizard.impl.DBWizardState$1$1, DEBUG, DB component setHost: p5wiadbs10.FNFIS.COM/10.132.205.63
        • 1. Re: DR server still showing consolidation users and messages from production
          beyerch2
          Hi,

          Interestingly enough, I had similar, though not exact, issue building an "all-in-one" VM System 11 (11.1.2.1) environment. Installed "everything" (SQL Server, foundation servers, Workspace, HFM, FDM, EPMA, Calc Mgr, FR, Smartview, LCM, etc, etc, etc). It worked fine. I copied the VM to another computer and parts of it stopped working for me. The issue turned out that in some parts of the databases / config files, it didn't store the hostname it had actually resolved it and stored IP address. (In my case though, workspace would not even work so my situation seems a bit different)

          While I hope this doesn't come off as some sort of self-promotion, on my linkedin profile, I have my slides from Kscope where I actually walked through how I fixed the VM issue. Perhaps my demo will help you find something that you may have overlooked? (I also discuss some tools earlier in the powerpoint that you can use to make the searching simpler)

          http://www.linkedin.com/in/charlesbeyer

          Look for the Kscope 12 folder in the "Box" app on my profile page. I believe it is at the bottom of the page. If you can't find it, let me know and I'll dig up the directly DL URL.

          Thanks
          • 2. Re: DR server still showing consolidation users and messages from production
            958766
            I see the box but I dont think I am seeing the DR document.

            What is it called or what is the direct linke.
            • 3. Re: DR server still showing consolidation users and messages from production
              beyerch2
              https://www.box.net/s/34d57fb45b8d4abba908

              The document is the Troubleshooting Powerpoint. I believe I walk through the IP address fixing around slide 68.
              • 4. Re: DR server still showing consolidation users and messages from production
                958766
                Thanks. The VM and what we did were somewhat different but helpful.

                When I ran the epm_registry batch file, it only found names.

                I tried to run a validate.bat, but it is giving me a null pointer exception. I will have to take a look at the documentation on it.

                We are running on windows servers, sql server database, and version 11.1.1.1.3 if I got all the 1 in the string correctly.

                I wrote a sql to scan all the tables for text type columns and search for the IP. It was only found in the log tables.

                I also scanned the application directory and the system directory on the shared services and hfm servers for the IP, it it was only found in a log file.

                Not sure where to go now.

                Thanks again.

                Otto
                • 5. Re: DR server still showing consolidation users and messages from production
                  beyerch2
                  Couple more comments :

                  1.) WHERE are you accessing HFM Web from? Locally on one of those servers or from another computer?

                  If this were me at this point, I would break out Fiddler2 (to verify which servers web traffic is coming/going to) and then I would fire up wireshark (all network scanning) to try and figure out what server(s) are trying to talk ot production and when. Network Monitor from MS may be a good option as well as I believe you can see network traffic by PROCESS which will help you narrow down what program(s) are trying to hit production envirionment information. (http://www.microsoft.com/en-us/download/details.aspx?id=4865)

                  My understanding is that HFM directly only has cluster information in the HFM database (HSX_CLUSTER_INFO, HSX_CLUTER_CONTROLLERS, HSX_CLUSTER_SERVERS, etc.) There shouldn't be any other machine specific information in HFM database, etc.

                  I do know that the workspace has some redirects going on in the .CONF file; however, if there was something hardcoded, you would have found that...

                  My guess is that something in the Shared Services/Foundation Services database is pointing to production as the other servers talk to that machine for config info. Otherwise, something is buried in a web config file, but you should have discovered that already.

                  2.) Additionally, note that HOSTS file changes only affect DNS resolution. If there are any type of broadcast messages or WINS resolution happening that won't address it. To hardcode WINS resolution, you would update LMHOSTS file in the same folder as HOSTS. Using wireshark to monitor network traffic on the same subnet as those servers would definitely be valuable here...... Broadcast messages would be using MAC address of networkcard and not hostname, for instance.

                  3.) Maybe try some simpler stuff first. What happens if you re-register the app through workspace? What about unregistering / re-registering server to shared services via configuration utility? (Though, I'd be a little bit concerned that it might unregister your real application if it is talking to production shared services...)

                  Edited by: beyerch2 on Aug 30, 2012 8:51 AM
                  • 6. Re: DR server still showing consolidation users and messages from production
                    958766
                    I added entries to the lmhosts and rebooted, it did not fix the issue.

                    I am also leary of adding and removing he applicaton per item 3 without knowing where the missing IP is being used for fear of effecting my production system.

                    I appreciate the help, any more suggestions would be appreciated.
                    • 7. Re: DR server still showing consolidation users and messages from production
                      958766
                      Now I am confused.
                      We have two workspace servers.
                      My link normally goes to link. Yesterday, I went to the other server to look at workspace.

                      One workspace server is showing prod.
                      The other workspace server is showing DR.

                      I am not sure now long this has been going on but I don't know if this brings any ideas to mind.

                      Thanks
                      Otto