8 Replies Latest reply: Jul 30, 2013 9:46 AM by JohnGoodwin RSS

    Testing failover in Essbase clustering

    866532

      Hello All,

       

      We are on Windows 2008 R2 64bit server and 11.1.2.1 EPM version. We have Active Passive Essbase cluster. I would like to do the test for fail-over.

      What ways I can initiate the failover to occur.

       

      As of now I just delete the Essbase.exe forcefully in the task manager. 

       

      thanks

        • 1. Re: Testing failover in Essbase clustering
          JohnGoodwin

          Seeing as it is windows has it been configured using Windows Server Failover Clustering?

           

          Cheers

           

          John

          http://john-goodwin.blogspot.com/

          • 3. Re: Testing failover in Essbase clustering
            JohnGoodwin

            Well can't you test from the Failover Cluster Manager or by command line?

             

            Cheers

             

            John

            http://john-goodwin.blogspot.com/

            • 4. Re: Testing failover in Essbase clustering
              866532

              Thanks John.

               

              Do you mean by "Move this service or application to another node" option in the Failover Cluster Manager ? I am able to move the resource from from one node to the other. But would that be a realistic test ?

              I want to recreate a scenario, when Essbase.exe process crashes. The resource fails over to the different node.

               

              Also do you think, if DNS stops working, that might crash essbase.exe process ?

              • 5. Re: Testing failover in Essbase clustering
                JohnGoodwin

                Yes I mean by moving it over to another node as this will confirm the failover configuration is working, the cluster will be configured to monitor the opmn service so you can try stopping it to see if the failover occurs correctly.

                 

                Cheers

                 

                John

                http://john-goodwin.blogspot.com/

                • 6. Re: Testing failover in Essbase clustering
                  866532

                  Can someone shed some light on 11.1.2.1 fail-over on Windows server. Why I can't initiate fail-over using command line ?

                   

                  Below are my Test for ESSBASE failover:

                  Environment: Windows 2008 R2 64bit

                  Essbase version : 11.1.2.1.000

                   

                  1. Moving Essbase resource from one node to another node using Microsoft Cluster Manager --- SUCCESSFUL

                  2. Using OPMNCTL command --- UNSUCCESSFUL

                   

                  Steps for command line:

                  a.  Start Essbase on Node 1 (active) using:

                  opmnctl start
                  opmnctl startproc ias-component=Essbase1

                   

                  b. Start OPMN on Node 2:

                  opmnctl start

                       Confirm Essbase is Down on Node 2:

                  opmnctl status

                   

                  c. On Node 1, issue the command:

                  opmnctl stopproc ias-component=Essbase1

                   

                  Essbase should fail-over to the second node, but it didn't.

                   

                  3. Forcefully killing Essbase.exe - UNSUCCESSFUL

                  The OPMN managing Essbase.exe starts the process again. Not sure why is it doing so, I have the below setting.

                  <process-set id="AGENT" restart-on-death="false">

                   

                  4. Forcefully killing opmn.exe managing Essbase.exe --- UNSUCCESSFUL

                  OPMN.exe and Essbase.exe starts back again. The Parent OPMN.exe might restarting them.

                   

                  5. Forcefully killing Parent OPMN.exe ---UNSUCCESSFUL

                  Everything starts back again on the same node. Not sure what is starting this now.

                   

                  6. Disabling Network adapter on the Node1. ---SUCCESSFUL

                  Failover successful from node1 to node2.

                   

                  NODE1

                  <?xml version="1.0" encoding="UTF-8"?><opmn xmlns="http://www.oracle.com/ias-instance">

                     <log path="D:\Oracle\Middleware\user_projects\epmsystem9\diagnostics\logs\OPMN\opmn\opmn.log" comp="internal;ons;pm" rotation-size="1500000"/>

                     <debug path="D:\Oracle\Middleware\user_projects\epmsystem9\diagnostics\logs\OPMN\opmn\opmn.dbg" comp="internal;ons;pm" rotation-size="1500000"/>

                     <notification-server interface="any">

                        <ipaddr remote="esbt1"/>

                        <port local="6711" remote="6712"/>

                        <ssl enabled="true" wallet-file="D:\Oracle\Middleware\user_projects\epmsystem9\config\OPMN\opmn\wallet"/>

                         <topology>

                          <nodes list="esbt1:6712,esbt2:6712"/>

                      </topology>

                      </notification-server>

                     <process-manager>

                        <process-modules>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnohs">

                              <module-id id="OHS2"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmncustom">

                              <module-id id="CUSTOM"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnoid">

                              <module-id id="OID"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnbi">

                              <module-id id="BIServer"/>

                              <module-id id="BIScheduler"/>

                              <module-id id="BIPresentationServices"/>

                              <module-id id="BIClusterController"/>

                              <module-id id="BIJavaHost"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnovd">

                              <module-id id="OVD"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnwc">

                              <module-id id="WebCache"/>

                              <module-id id="WebCache-admin"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnreports">

                              <module-id id="ReportsServices"/>

                              <module-id id="ReportsBridgeServices"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmndisco">

                              <module-id id="Disco_ServicesStatus"/>

                              <module-id id="Disco_SessionServer"/>

                              <module-id id="Disco_PreferenceServer"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnemagent">

                              <module-id id="EMAGENT"/>

                           </module>

                        <module path="$ORACLE_HOME/opmn/lib/libopmness"><module-id id="ESS"/></module></process-modules>

                        <ias-instance id="EPM_epmsystem9" name="EPM_epmsystem9">

                          <environment>

                             <variable id="TEMP" value="C:\Users\HYP11S~1\AppData\Local\Temp\2"/>

                             <variable id="TMP" value="C:\Users\HYP11S~1\AppData\Local\Temp\2"/>

                          </environment>

                        <ias-component id="EssbasetestCluster-1">

                    <process-type id="EssbaseAgent" module-id="ESS" service-failover="1" service-weight="101">

                      <environment>

                        <variable id="EPM_ORACLE_HOME" value="D:\Oracle\Middleware\EPMSystem11R1"/>

                        <variable id="EPM_ORACLE_INSTANCE" value="D:\Oracle\Middleware\user_projects\epmsystem9"/>

                        <variable id="ESSBASEPATH" value="$EPM_ORACLE_HOME/products/Essbase/EssbaseServer"/>

                        <variable id="ARBORPATH" value="E:\Oracle\Middleware\user_projects\epmsystem9\EssbaseServer\essbaseserver1"/>

                        <variable id="ESSBASEEXE" value="$ESSBASEPATH/bin/ESSBASE"/>

                        <variable id="HYPERION_LOGHOME" value="$EPM_ORACLE_INSTANCE/diagnostics/logs/essbase"/>

                        <variable id="HYPERION_HOME" value="$EPM_ORACLE_HOME"/>

                        <variable id="ORA_NLS10" value="$EPM_ORACLE_HOME/nls/data"/>

                        <variable id="ESSLANG" value="English_UnitedStates.Latin1@Binary"/>

                        <variable id="ESS_JVM_OPTION1" value="-Xmx256M"/>

                        <variable id="ESS_CSS_JVM_OPTION4" value="-Xms256M" />

                        <variable id="ESS_CSS_JVM_OPTION5" value="-Xmx1024M" />

                        <variable id="ESS_CSS_JVM_OPTION6" value="-Doracle.dms.context=OFF"/>

                        <variable id="ESS_CSS_JVM_OPTION7" value="-Djava.util.logging.config.class=oracle.core.ojdl.logging.LoggingConfiguration"/>

                        <variable append="true" id="PATH" value="$EPM_ORACLE_HOME/common/ODBC-64/Merant/6.0/lib"/>

                        <variable append="true" id="PATH" value="$ESSBASEPATH/bin"/>

                        <variable append="true" id="PATH" value="$EPM_ORACLE_HOME/bin"/>

                        <variable id="ODBCLIBHOME" value="$EPM_ORACLE_HOME/common/ODBC-64/Merant/6.0/lib"/>

                        <variable id="ODBCINI" value="$EPM_ORACLE_HOME/common/ODBC-64/Merant/6.0/odbc.ini"/>

                        <variable id="ODBCINST" value="$EPM_ORACLE_HOME/common/ODBC-64/Merant/6.0/odbcinst.ini"/>

                        <variable id="OPMN_MANAGED_AGENT" value="true"/>

                        <variable id="ORACLE_THREAD_STACKSIZE" value="1048576"/>

                      </environment>

                      <start retry="2" timeout="600"/>

                      <stop timeout="600"/>

                      <restart retry="2" timeout="600"/>

                      <port id="essbase-port-range" range="32768-33768"/>

                      <process-set id="AGENT" restart-on-death="false">

                        <module-data>

                          <category id="start-parameters">

                            <data id="start-executable" value="$ESSBASEEXE"/>

                            <data id="agent-port" value="1423"/>

                            <data id="agent-secure-port" value="6423"/>

                            <data id="start-args" value="-b"/>

                          </category>

                          <category id="restart-parameters">

                            <data id="restart-executable" value="integrator"/>

                          </category>

                          <category id="ping-parameters">

                            <data id="ping-interval" value="20"/>

                            <data id="ping-timeout" value="20"/>

                          </category>

                        </module-data>

                      </process-set>

                    </process-type>

                  </ias-component></ias-instance>

                        <rmd-definitions>

                   

                  NODE2

                  <?xml version="1.0" encoding="UTF-8"?><opmn xmlns="http://www.oracle.com/ias-instance">

                     <log path="D:\Oracle\Middleware\user_projects\epmsystem10\diagnostics\logs\OPMN\opmn\opmn.log" comp="internal;ons;pm" rotation-size="1500000"/>

                     <debug path="D:\Oracle\Middleware\user_projects\epmsystem10\diagnostics\logs\OPMN\opmn\opmn.dbg" comp="internal;ons;pm" rotation-size="1500000"/>

                     <notification-server interface="any">

                        <ipaddr remote="esbt2"/>

                        <port local="6711" remote="6712"/>

                        <ssl enabled="true" wallet-file="D:\Oracle\Middleware\user_projects\epmsystem10\config\OPMN\opmn\wallet"/>

                         <topology>

                          <nodes list="esbt1:6712,esbt2:6712"/>

                      </topology>

                      </notification-server>

                     <process-manager>

                        <process-modules>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnohs">

                              <module-id id="OHS2"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmncustom">

                              <module-id id="CUSTOM"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnoid">

                              <module-id id="OID"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnbi">

                              <module-id id="BIServer"/>

                              <module-id id="BIScheduler"/>

                              <module-id id="BIPresentationServices"/>

                              <module-id id="BIClusterController"/>

                              <module-id id="BIJavaHost"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnovd">

                              <module-id id="OVD"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnwc">

                              <module-id id="WebCache"/>

                              <module-id id="WebCache-admin"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnreports">

                              <module-id id="ReportsServices"/>

                              <module-id id="ReportsBridgeServices"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmndisco">

                              <module-id id="Disco_ServicesStatus"/>

                              <module-id id="Disco_SessionServer"/>

                              <module-id id="Disco_PreferenceServer"/>

                           </module>

                           <module path="$ORACLE_HOME/opmn/lib/libopmnemagent">

                              <module-id id="EMAGENT"/>

                           </module>

                        <module path="$ORACLE_HOME/opmn/lib/libopmness"><module-id id="ESS"/></module></process-modules>

                        <ias-instance id="EPM_epmsystem10" name="EPM_epmsystem10">

                          <environment>

                             <variable id="TEMP" value="C:\Users\HYP11S~1\AppData\Local\Temp\2"/>

                             <variable id="TMP" value="C:\Users\HYP11S~1\AppData\Local\Temp\2"/>

                          </environment>

                        <ias-component id="EssbasetestCluster-1">

                    <process-type id="EssbaseAgent" module-id="ESS" service-failover="1" service-weight="100">

                      <environment>

                        <variable id="EPM_ORACLE_HOME" value="D:\Oracle\Middleware\EPMSystem11R1"/>

                        <variable id="EPM_ORACLE_INSTANCE" value="D:\Oracle\Middleware\user_projects\epmsystem10"/>

                        <variable id="ESSBASEPATH" value="$EPM_ORACLE_HOME/products/Essbase/EssbaseServer"/>

                        <variable id="ARBORPATH" value="E:\Oracle\Middleware\user_projects\epmsystem9\EssbaseServer\essbaseserver1"/>

                        <variable id="ESSBASEEXE" value="$ESSBASEPATH/bin/ESSBASE"/>

                        <variable id="HYPERION_LOGHOME" value="$EPM_ORACLE_INSTANCE/diagnostics/logs/essbase"/>

                        <variable id="HYPERION_HOME" value="$EPM_ORACLE_HOME"/>

                        <variable id="ORA_NLS10" value="$EPM_ORACLE_HOME/nls/data"/>

                        <variable id="ESSLANG" value="English_UnitedStates.Latin1@Binary"/>

                        <variable id="ESS_JVM_OPTION1" value="-Xmx256M"/>

                        <variable id="ESS_CSS_JVM_OPTION4" value="-Xms256M" />

                        <variable id="ESS_CSS_JVM_OPTION5" value="-Xmx1024M" />

                        <variable id="ESS_CSS_JVM_OPTION6" value="-Doracle.dms.context=OFF"/>

                        <variable id="ESS_CSS_JVM_OPTION7" value="-Djava.util.logging.config.class=oracle.core.ojdl.logging.LoggingConfiguration"/>

                        <variable append="true" id="PATH" value="$EPM_ORACLE_HOME/common/ODBC-64/Merant/6.0/lib"/>

                        <variable append="true" id="PATH" value="$ESSBASEPATH/bin"/>

                        <variable append="true" id="PATH" value="$EPM_ORACLE_HOME/bin"/>

                        <variable id="ODBCLIBHOME" value="$EPM_ORACLE_HOME/common/ODBC-64/Merant/6.0/lib"/>

                        <variable id="ODBCINI" value="$EPM_ORACLE_HOME/common/ODBC-64/Merant/6.0/odbc.ini"/>

                        <variable id="ODBCINST" value="$EPM_ORACLE_HOME/common/ODBC-64/Merant/6.0/odbcinst.ini"/>

                        <variable id="OPMN_MANAGED_AGENT" value="true"/>

                        <variable id="ORACLE_THREAD_STACKSIZE" value="1048576"/>

                      </environment>

                      <start retry="2" timeout="600"/>

                      <stop timeout="600"/>

                      <restart retry="2" timeout="600"/>

                      <port id="essbase-port-range" range="32768-33768"/>

                      <process-set id="AGENT" restart-on-death="false">

                        <module-data>

                          <category id="start-parameters">

                            <data id="start-executable" value="$ESSBASEEXE"/>

                            <data id="agent-port" value="1423"/>

                            <data id="agent-secure-port" value="6423"/>

                            <data id="start-args" value="-b"/>

                          </category>

                          <category id="restart-parameters">

                            <data id="restart-executable" value="integrator"/>

                          </category>

                          <category id="ping-parameters">

                            <data id="ping-interval" value="20"/>

                            <data id="ping-timeout" value="20"/>

                          </category>

                        </module-data>

                      </process-set>

                    </process-type>

                  </ias-component></ias-instance>

                        <rmd-definitions>

                  • 7. Re: Testing failover in Essbase clustering
                    JohnGoodwin

                    Using Windows Server Failover Clustering bases the failover on the OPMN service and not the Essbase service, it is not the same as OPMN failover, if the OPMN processes are all down then in theory it should failover.

                     

                    Cheers

                     

                    John

                    http://john-goodwin.blogspot.com/

                    • 8. Re: Testing failover in Essbase clustering
                      866532

                      So basically it means, that if ESSBASE.exe crashes for some reason on active node it wont fail-over to the passive node.

                       

                      The other question is when I tried killing all OPMN.exe processes on active node, why it didnt failover to passive node.