1 2 Previous Next 18 Replies Latest reply on Jan 2, 2019 6:34 AM by Chris Badenhorst

    Hyperion Financial Management Consolidation Aborts

    Chris Badenhorst

      Hi

       

      Our HFM consolidation aborts intermittently. We have been having this issue for the past 2 years after upgrading to 11.1.2.4. Checking the logs we see errors "Error executing VBScript" with the line number. But nothing further. A POV will consolidate fine, later it aborts, then OK again etc. And the line number differs. We have been applying latest patches etc. and have just given up. Until I came across the DOC ID 2347940.1 today.

       

      The symptoms are exactly as set out in DOC ID 2347940.1. Has anyone experienced this on Windows Server?

       

      Thank you.

       

      Chris.

        • 1. Re: Hyperion Financial Management Consolidation Aborts
          CBarbieri

          This is almost certainly a rules error. Go into consolidation administration > system messages, and look at the details of all the errors (often around 4 errors will occur) at the time the consolidation aborted. In at least one errors you may see a reference to a metadata member that doesn't exist, or an invalid intersection, or a divide by zero, type mismatch, or something else. If you want more help on that, paste the contents of all the messages here.

           

          Regarding the document you linked: are you running HFM on Exalytics? The document describes a bug specific to that platform.

           

          - Chris

          • 2. Re: Hyperion Financial Management Consolidation Aborts
            Chris Badenhorst

            Hi

             

            This is not a rules error. There is nothing wrong with the syntax and the lines are not referring to invalid members. Multiple experts have looked at our rules file and cannot find a problem there.

             

            It doesn't give a reference to a member like it would with a genuine rules error.

             

            The issue is intermittent and it works again after reboot. A syntax error will cause consistent vb script errors and aborts on the same line. We had the issue for the past 2 years but just left it as it was intermittent. But after coming across the document I'm engaging with Oracle Support.

            • 3. Re: Hyperion Financial Management Consolidation Aborts
              CBarbieri

              Can you paste the sequence of error messages here so we can help you? Also, can you confirm whether you're running HFM on Exalytics?

               

              Just for discussion, there are reasons why rules errors can be intermittent:

              1) Data driven rules: can run based on data conditions. If the number in a particular cell is X then do something...   Errors related to this may occur sporadically depending on the data conditions.

              2) ImpactStatus: this might fail if the target is locked, which can be sporadic depending on the events

              3) Review level status driven rules: read the review level of a POV and perform an action. Again, the sequence of events to start, promote, etc. can vary over time causing rules to vary.

              4) On error resume next: this tells HFM that in the event a vbscript error occurs, just continue.

              5) HS.IsValidDest function: this will write an error message to the log that the destination is invalid, but the rule would continue.

              6) WriteToFile: under some condition HFM tries to write to a file or access a file system location that it can't reach. If you have multiple servers and each HFM application server is configured with a different service account, maybe only one of the service accounts can access the target location but the other one can't. In this situation you can look to see if the errors always occur on one server and not the other.

               

              This is off the top of my head, but it should illustrate that there can be scientific reasons for an intermittent rules failure.

               

              Of course this leaves room for a bug or something else, but the logs ought to offer some details that leads us down a path to investigate, if the talented people on this forum can help you in any way.

               

              - Chris

              • 4. Re: Hyperion Financial Management Consolidation Aborts
                Chris Badenhorst

                Hi

                 

                The most likely cause would be 1.

                 

                We have never used On error resume next, maybe I should try that But then there is a big risk of an error not being picked up.

                 

                We are running HFM on Microsoft SQLServer - 2012 and Microsoft Windows x64 (64-bit) - 2012 R2

                 

                Here are the lines per the ODL log. I've replaced the server, IP address, app, entity name and User ID with xxx.

                 

                [2018-09-24T04:44:52.712638+02:00] [XFM] [TRACE:1] [EPMHFM-01546] [XFM] [ecid: 0000MNxvqUw6aMOLyiXBiW1RdLk1000000] [File: CHsvDSCalculate.cpp] [Line: 14657] [userId: ] [Msg arguments: ] [appName: xxx] [pid: 7108] [tid: 9392] [host: xxx] [nwaddr: xxx:0;] [errorCode: 41C81] [srcException: NotSpec] [errType: 1] [dbUpdate: 1] [11.1.2.4.205.5274] Consolidation started. [[]]

                [2018-09-24T04:46:37.417314+02:00] [XFM] [ERROR:1] [EPMHFM-00102] [XFM] [ecid: 0000MNxvqUw6aMOLyiXBiW1RdLk1000000] [File: CHsvDSCalculate.cpp] [Line: 5211] [userId: ] [Msg arguments: ] [appName: xxx] [pid: 7108] [tid: 9392] [host: xxx] [nwaddr: xxx:0;] [errorCode: 80042FC2] [srcException: XfmExc] [errType: 0] [dbUpdate: 0] [11.1.2.4.205.5274] Error executing VBScript  [[]]

                [2018-09-24T04:46:37.417314+02:00] [XFM] [ERROR:1] [EPMHFM-00102] [XFM] [ecid: 0000MNxvqUw6aMOLyiXBiW1RdLk1000000] [File: CHsvDSCalculate.cpp] [Line: 14926] [userId: ] [Msg arguments: ] [appName: xxx] [pid: 7108] [tid: 9392] [host: xxx] [nwaddr: xxx:0;] [errorCode: 80042FC2] [srcException: XfmExc] [errType: 0] [dbUpdate: 0] [11.1.2.4.205.5274] Error executing VBScript  [[]]

                [2018-09-24T04:46:37.432919+02:00] [XFM] [TRACE:1] [EPMHFM-02402] [XFM] [ecid: 0000MNxvqUw6aMOLyiXBiW1RdLk1000000] [File: CHsvDSCalculate.cpp] [Line: 15176] [userId: ] [Msg arguments: ] [appName: xxx] [pid: 7108] [tid: 9392] [host: xxx] [nwaddr: xxx:0;] [errorCode: 41C82] [srcException: NotSpec] [errType: 1] [dbUpdate: 1] [11.1.2.4.205.5274] Consolidation finished. [[]]

                [2018-09-24T04:46:37.432919+02:00] [XFM] [ERROR:1] [EPMHFM-06876] [XFM] [ecid: 0000MNxvqUw6aMOLyiXBiW1RdLk1000000] [File: CHsvDSCalculate.cpp] [Line: 15185] [userId: ] [xxx;id:1902722782,BUDGET_OPS,2019,.xxx,January-December, Consolidation type:ALLWITHDATA] [appName: xxx] [pid: 7108] [tid: 9392] [host: xxx] [nwaddr: xxx:0;] [errorCode: 80041C92] [srcException: NotSpec] [errType: 1] [dbUpdate: 1] [11.1.2.4.205.5274] xxx Consolidation Aborted id:1902722782,BUDGET_OPS,2019,.xxx,January-December, Consolidation type:ALLWITHDATA [[]]

                [2018-09-24T04:46:37.432919+02:00] [XFM] [ERROR:1] [EPMHFM-00102] [XFM] [ecid: 0000MNxvqUw6aMOLyiXBiW1RdLk1000000] [File: CXfmCalculate.cpp] [Line: 3856] [userId: xxx] [Msg arguments: ] [appName: xxx] [pid: 7108] [tid: 9392] [host: xxx] [nwaddr: xxx:0;] [errorCode: 80042FC2] [srcException: NotSpec] [errType: 1] [dbUpdate: 1] [11.1.2.4.205.5274] Error executing VBScript  [[Consolidation: BUDGET_OPS,2019,January-December]]

                [2018-09-24T04:46:37.432919+02:00] [XFM] [ERROR:1] [EPMHFM-00102] [XFM] [ecid: 0000MNxvqUw6aMOLyiXBiW1RdLk1000000] [File: CXfmCalculate.cpp] [Line: 1524] [userId: ] [Msg arguments: ] [appName: xxx] [pid: 7108] [tid: 9392] [host: xxx] [nwaddr: xxx:0;] [errorCode: 80042FC2] [srcException: XfmExc] [errType: 0] [dbUpdate: 0] [11.1.2.4.205.5274] Error executing VBScript  [[]]

                [2018-09-24T04:46:37.432919+02:00] [XFM] [ERROR:1] [EPMHFM-00102] [XFM] [ecid: 0000MNxvqUw6aMOLyiXBiW1RdLk1000000] [File: ..\Handlers\data\DataHandlerServiceInternal.cpp] [Line: 6214] [userId: ] [Msg arguments: ] [appName: xxx] [pid: 7108] [tid: 9392] [host: xxx] [nwaddr: xxx:0;] [errorCode: 80042FC2] [srcException: XfmExc] [errType: 1] [dbUpdate: 1] [11.1.2.4.205.5274] Error executing VBScript  [[]]

                 

                Here is one of the rules it sometimes aborts on. The line is: If Trans_Adj <> 0 Then HS.Exp "A#" & vAccount & ".I#" & vICP & ".C1#Translation_Adj.C2#" & vCustom2 & ".C3#" & vCustom3 & ".C4#" & vCustom4 & " = " & Trans_Adj

                 

                Sometimes I get an extra line in the ODL log: [2018-09-26T21:16:27.343262+02:00] [XFM] [ERROR:1] [EPMHFM-02764] [XFM] [ecid: ] [File: CHsvDSRulesOM.cpp] [Line: 8273] [userId: ] [A#CF_OIA.I#[ICP None].C1#Translation_Adj.C2#[None].C3#[None].C4#[None]] [appName: xxx] [pid: 27120] [tid: 17268] [host: xxx] [nwaddr: xxx:0;] [errorCode: 8004282C] [srcException: XfmExc] [errType: 1] [dbUpdate: 1] [11.1.2.4.205.5274] Invalid number of operators specified. [[]]

                 

                Below is an extract of the bigger rule.

                 

                Set DataUnit = HS.OpenDataUnit("A{CF_NetCCE.[BASE]}")

                   NumItems = DataUnit.GetNumItems

                   For item_loop = 0 To NumItems - 1  

                   Call DataUnit.GetItem(item_loop, vAccount, vICP, vCustom1, vCustom2, vCustom3, vCustom4, vData)

                   If vData <> 0 Then

                    If HS.Account.IsDescendant("CF_NetCCE",vAccount) Then

                      'Exclude Certain Accounts from the Adjustment

                      Select Case vAccount

                      Case "CF_DPDMS_OA", "CF_DPDMS", "CF_Rest_Cash"

                       'Do nothing for these accounts

                      Case Else

                       'Determine if Custom1 Translation_Adj has already been calculated for this account

                       Translation_Adj_Value = HS.GetCell("A#" & vAccount & ".C1#Translation_Adj.C2#[None].C3#[None].C4#[None].I#" & vICP)

                       If Translation_Adj_Value = 0 Then

                        If HS.Account.IsDescendant("CF_OXM",vAccount) Then 'The sum of these adjustments should be allocated to the CF_IRBO Cashflow account.

                      

                          'Determine if Custom1 Translation_Adj routine has already been executed for the account/ICP and allocated to the CF_IRBO Cashflow account

                          If Not vAccount = vAccount_Prev Or Not vICP = vICP_Prev Then

                      

                           ECT_Value = HS.GetCell("A#" & vAccount & ".C1#CashFlow.C2#[None].C3#[None].C4#[None].I#" & vICP & ".V#<Entity Curr Total>")

                           PC_Value = HS.GetCell("A#" & vAccount & ".C1#CashFlow.C2#[None].C3#[None].C4#[None].I#" & vICP)

                           ECT_Value_Translated_EBITDA = ECT_Value * EBITDA_Rate

                           Trans_Adj = ECT_Value_Translated_EBITDA - PC_Value

                           Total_Adj_CF_OXM_Descendants = Total_Adj_CF_OXM_Descendants + Trans_Adj

                           vAccount_Prev = vAccount

                           vICP_Prev = vICP

                        

                          End If

                      

                        Else 'Apply all other Adjustments to the specific account              

                          ECT_Value = HS.GetCell("A#" & vAccount & ".C1#CashFlow.C2#[None].C3#[None].C4#[None].I#" & vICP & ".V#<Entity Curr Total>")

                          PC_Value = HS.GetCell("A#" & vAccount & ".C1#CashFlow.C2#[None].C3#[None].C4#[None].I#" & vICP)

                          ECT_Value_Translated_EBITDA = ECT_Value * EBITDA_Rate

                      

                          Trans_Adj = Round((ECT_Value_Translated_EBITDA - PC_Value),2)

                      

                          If Trans_Adj <> 0 Then HS.Exp "A#" & vAccount & ".I#" & vICP & ".C1#Translation_Adj.C2#" & vCustom2 & ".C3#" & vCustom3 & ".C4#" & vCustom4 & " = " & Trans_Adj   

                     

                        End If 'HS.Account.IsDescendant("CF_OXM",vAccount)

                       End If 'Translation_Adj_Value <> 0

                      End Select

                      Trans_Adj = 0

                    End If 'HS.Account.IsDescendant("Validations",vAccount)

                   End If

                  Next

                 

                All variables such as EBITDA_Rate is defined etc. but I did not paste that here.

                 

                This rule is not new and worked perfectly in 11.1.2.3.

                 

                Any help is welcome.

                • 5. Re: Hyperion Financial Management Consolidation Aborts
                  CBarbieri

                  ok. So this is a Wintel machine therefore the Exalytics issue doesn't apply.

                   

                  Regarding the messages, I'm going to focus on two things you shared: that the error comes from a line that uses HS.EXP to set a single numeric value to an intersection, and the message " Invalid number of operators specified." It doesn't make sense to me, but what happens if Trans_Adj is null? My problem is that the variable ought to be zero or it will have a number, but what happens if it's null? then the expression would set an intersection equal to a null value. One idea is to use If ABS(Trans_Adj) > 0 which means it has to have a value, rather than If Trans_Adj <> 0. Another is to replace the HS.EXP line with a HS.SetDataWithPOV function. In both cases you are writing a single data point back to a defined intersection. The difference is that SetDataWithPOV always expects a data value whereas HS.EXP usually expects an intersection.

                   

                  Syntax: HS.SetDataWithPOV "TargetPOV", Data, True or False

                  True if you want the values going to the target to accumulate. This is useful if you are doing an allocation where multiple expenses go to a single allocation account.

                  False if this is merge mode, which is the default. Last value written to a cell is the value you get.

                  Another useful function is HS.GetCellNoData which returns two variables: the data value, plus a True if the cell is empty, or False if the cell has something in it.

                  dNumber = HS.GetCellNoData("SourcePOV", booleanVariable)

                   

                  If booleanVariable = False And Abs(dNumber) > 0.01 Then

                  ...

                   

                  Often I don't care what the number is, I only care if there IS something in the cell. This can be useful.

                   

                  These are just some ideas you can try with low risk. I agree that as written it ought to work. If you can reproduce the error, you could use a write to file to debug this. If you haven't used WriteToFile to debug rules before, send me an email cbarbieri@finit.com and I'll send you a presentation that walks you through the steps.

                   

                   

                  In my opinion I would pursue this as a rules issue. There are generally only two reasons to get a VBScript error: rules or custom member lists.

                   

                  - Chris

                   

                  1 person found this helpful
                  • 6. Re: Hyperion Financial Management Consolidation Aborts
                    Chris Badenhorst

                    Hi

                     

                    Many thanks for the possible solutions. Will apply and let you know the outcome.

                     

                    Chris.

                    • 7. Re: Hyperion Financial Management Consolidation Aborts
                      Chris Badenhorst

                      Hi

                       

                      I've applied your suggestions. ABS() didn't do anything. Still experienced the same issue on same random lines. However, replacing HS.EXP function with HS.SetDataWithPOV did help, but now it just aborts on different lines. So I'll keep on replacing the HS.EXP function and the line in the vb script error will just be different - lines that never had an issue before.

                       

                      This is really frustrating.

                       

                      Do you have details on the bug published in Doc ID 2347940.1?

                       

                      To add. It is worse when the system is busy e.g. lots of users, maybe doing data loads, maybe a data extract and running a consolidation. Also if consolidations are run in parallel on independent scenarios. We experience this more when we are doing budgets over multiple months. Actuals we are OK but do get the an error now and again. It's very weird.

                      • 8. Re: Hyperion Financial Management Consolidation Aborts
                        Chris Badenhorst

                        Hi

                         

                        Further update. We rebooted yesterday morning and now no errors. Ran same consols multiple times. Expect the errors to be back again in next couple of days. Will then do a reboot again.

                        • 9. Re: Hyperion Financial Management Consolidation Aborts
                          Jeo123

                          Do you have multithreading consolidations turned on?  If you're saying this is new to 11.1.2.4, that's by far the biggest change.  You could be looking at a problem that happens due to the entity calculation order changing.

                           

                          Also, is this properly value restricted?  I noticed you reference PC_Value, but don't specify a Value, so you're relying on whatever is currently being calculated.  Is there a value restriction somewhere else in the rules to control this?

                          • 10. Re: Hyperion Financial Management Consolidation Aborts
                            Chris Badenhorst

                            Hi

                             

                            It will only execute the sub if the Value dimension member is a translated currency member.

                             

                            We have the following in the rules: If Not HS.Value.IsTransCur Then Exit Sub

                             

                            Will see what happens if multithreading is set to 0. But I'm concerned that our consolidations might be very slow then.

                             

                            Chris

                            • 11. Re: Hyperion Financial Management Consolidation Aborts
                              Jeo123

                              It will likely slow it down, but if it resolves your issue that will point you in the right direction.  Multi-threading can sometimes change the calculation order of entities, so if your aborted consolidations stop occuring, that means you have an order of operations problem and can try to look into that.

                               

                              For example, we had a problem where a hierarchy of EUR base with an EUR Parent was looking for the USD amount from the base entity.  With single threading, the USD main hierarchy always consolidated first, but we ran into a problem when multithreading would sometimes calculate the EUR parent before the USD Hierarchy caused the translation.

                               

                              Also, I'm not sure if it's your problem, but keep in mind that if the base and parent entities are the same currency, then when <Parent Currency> is calculated, HS.Value.IsTransCur is false.  That can occasionally burn people.

                              1 person found this helpful
                              • 12. Re: Hyperion Financial Management Consolidation Aborts
                                Chris Badenhorst

                                Hi

                                 

                                We have changed Multithreading to 0 in Production. The consolidations actually seem to be faster. In the past it used to stay 'stuck' for a long time in the progress bar on one of the higher sub-groups, which is no longer the case.

                                 

                                What you have said makes a lot of sense to me. When we got an abort on consolidating of the top parent, we then would consolidate each entity from the bottom up and then it will be OK.

                                 

                                If we don't get any aborts in the next 5 days, I'm sure Multithreading was the issue, and I will then mark your answer as correct.

                                 

                                Many thanks for your help on something that we have been having an issue with the past 2 years!

                                 

                                Chris

                                • 13. Re: Hyperion Financial Management Consolidation Aborts
                                  Chris Badenhorst

                                  Hi

                                   

                                  Just some feedback. Our aborts are back again this morning. 3 consecutive aborts on different scenarios. Immediately after ran them again and they were OK.

                                   

                                  If issue persists, will restart services.

                                   

                                  Chris

                                  • 14. Re: Hyperion Financial Management Consolidation Aborts
                                    Jeo123

                                    This definitely sounds like an order of operations problem, but I can't see anything about the line you originally said was the error that would be the cause.

                                     

                                    What line did it abort on this time? I know you referenced common lines, but knowing the exact one for this might shed some light on the problem.

                                     

                                    Also, are you able to replicate this in a test environment?  Doing that would let you start putting logging rules in place to isolate the problem further.

                                    1 2 Previous Next