Forum Stats

  • 3,839,756 Users
  • 2,262,532 Discussions
  • 7,901,052 Comments

Discussions

EPM 11.2.6 HFM Performance Bug

1235789

Answers

  • BajoD
    BajoD Member Posts: 5 Blue Ribbon

    Does anyone know is the patch officially released?

  • EPMTek
    EPMTek Member Posts: 14 Bronze Badge
    edited Mar 15, 2022 12:08PM

    As mentioned above, Dev patch is not available to all customers.

    Word on the street is official release is 16th March but don't quote me..... Question is whether patches for 11.2.7/11.2.8 will now arrive before 11.2.9.

    Reg Key, UseCalcstatusCache=1 plus the 11.2.6 patch take my test consolidation (A Weekly Scenario for a whole Year) from 60 mins down to 24 mins which is now 5 minutes faster than the 11.1.2.4 equivalent on the same VM host specs. I also have all VMs temporarily on the same host to eliminate network issues. Your mileage may vary.

    Dialling NumConsolidationThreads back to 12 on a 12-vCPU dedicated consolidation server has now improved things further (was at 24 for earlier testing)

    (UPDATE: 24 NumConsolidationThreads was 24 mins, 12 NumConsolidationThreads is 17.7 minutes, 8 NumConsolidationThreads is also 24 minutes). 12 is the magic number on this architecture.

    For timings, I am running each test consolidation twice on the SAME HFM App Server, to ensure that all required sub-cubes are in RAM, so I am then taking the second run as the time. I also restart the application on the relevant server to ensure that I'm starting from a consistent baseline.

    This is when having EPM Maestro available is very handy for running a load of these consistently on a set of POVs (thanks Saquib!).

    Interestingly some of my 11.2.8 clients have said 11.2.8 vanilla is faster than 11.1.2.4 so I think there has to be an application and/or subtle infra element to this also...

    Patch WL Admin, HFM App/Java servers AND don't forget to copy updated JAR to Payara if using 3rd-party tools like Merlin/Maestro/ReportAuthority etc.

    Looks like I can breathe out now - the client's UAT is now in danger of happening! :-). Roll on 16th March for the official 11.2.6 patch being on general release, as I hope that it will help a lot of people out!

    ** My testing has been Consolidation times only so far, so other HFM operations have not been tested yet **

    Kudos to the Oracle EPM Devs for tracking this down - obviously can't have been easy.

    Steve

    Thanos A
  • BajoD
    BajoD Member Posts: 5 Blue Ribbon

    Thanks for your update Steve. FYI I have seen improvement in consolidation run with 24 CPU's (about 10%) but I had a better run in 11.1.2.4 release with 8 CPU's.

  • EPMTek
    EPMTek Member Posts: 14 Bronze Badge
    edited Mar 15, 2022 3:30PM

    I have had some serious Consolidation time improvements by now changing the ConsolidationMultiThreadingScheme from 2 to 6, with my best time now coming down to 12.5 minutes (from original 60 minutes mentioned above - NumConsolidationThreads=12 on a 12-vCPU dedicated Consolidation Server).

    This could be application-related of course, but give it a try and see how you get on - I will try 3 (2+1) and 7 (4+2+1) in the interests of thoroughness (and as I have the system to myself for now). Won't bother with 5 (i.e. with 2-bit not set) as I am pretty sure it always needs parent multi-threading.

    The POV fits completely into RAM as there is no paging/LRU flushing going on, and I can see that the node counts don't vary in the System Messages detail - so whatever is happening is dependent on the HFM application consolidation code and must be CPU-bound.

    (UPDATE - ConsolidationMultiThreadingScheme = 3 gives 35 minutes, ConsolidationMultiThreadingScheme=7 gives 18.4 minutes on same POV).

    (UPDATE - ConsolidationMultiThreadingScheme = 6, NumConsolidationThreads=24 (12vCPU VM) gives 11.4 minutes, ConsolidationMultiThreadingScheme = 6, NumConsolidationThreads=36 (12vCPU VM) gives 9.8 minutes! I think I'm done now.)

    Interestingly, any ConsolidationMultiThreadingScheme setting with the 4-bit set seems to break the progress bar indicator in Running Tasks - Sev4 SR raised!))

    For background - from HFM settings:

    "Used for multiple settings related to how the consolidation process is run. Each setting can be enabled or disabled without affecting the other settings. You can choose more than one option and set the parameter to the sum of the chosen options. For example, to use options 2 and 4, set the parameter to 6. 1 - Allows the consolidation process to calculate and translate all children entities up to the [Parent] value before the numbers are consolidated into the parent entity. 2 - Allows the consolidation process to multi-thread parent-level entities, greatly improving the performance on large consolidations. 4 - Skips the initial calculation on all base-level entities. Those entities are calculated as needed when processing their parent entities. Can override at Application, Server or combination of both."

    This setting can be from 1-15, BUT nothing above the 4-bit is documented - does anyone have any insights to what the 8-bit might do....?

    *** I have NOT yet checked whether status/numbers are as expected - I am just after raw speed at the moment!!! ***

    At this rate it'll finish before it starts, and we'll finally have real-time Consolidations....;-)

    Assuming the 11.2.6.1 patch is indeed out tomorrow, then I will roll back the Dev patch and retest with my best settings to ensure that it wasn't all a dream - hope it works for you.

    Steve

  • BajoD
    BajoD Member Posts: 5 Blue Ribbon

    Thanks for your suggestions, forgot to mention that my platform is Windows 2019/ SQL Server 2016. What platform is your test done?

  • EPMTek
    EPMTek Member Posts: 14 Bronze Badge

    Windows Server 2019 + SQL Server 2019 (dedicated - i.e. not shared). All VMs on Nutanix, currently HFM App servers and SQL for this environment are residing on same host.

  • User_6TQOL
    User_6TQOL Member Posts: 17 Blue Ribbon

    Recently upgraded to HFM 11.2.6. Performance is much better than 11.1.2.4.

    Issues with ADF and also journal templates. Both bugs apparently fixed in 11.2.7

    Issues with Smart view formatting. This can be corrected by clear all format in excel.

    Web form lines are not straight

    Journal template error

    https://support.oracle.com/epmos/faces/DocumentDisplay?id=2831693.1

  • deeziel
    deeziel Member Posts: 6 Blue Ribbon

    Seem the 11.2.6.2 patch is now publicly available, haven't found it for other versions yet.



  • EPMTek
    EPMTek Member Posts: 14 Bronze Badge
    edited Mar 16, 2022 9:02AM

    Thanks for the heads-up Daniel. Here goes......;-)

    UPDATE1:

    HFM 11.2.6.2 patch applied BUT Registry Entry not applied. First run took 22.4 minutes/Second run 16.9 minutes on Consolidation Server (12 vCPU/NumConsolidationThreads=36, ConsolidationMultiThreadScheme=6) vs 9.8 minutes with Dev patch AND Registry Entry on same spec VM. >50% improvement therefore over no patch (60 mins) BUT not as fast as I have seen on the environment above (identical architecture but without Registry Entry).


    Let's see if the Reg Entry makes a difference.... Note that if you are using Merlin/Maestro/Report Authority that there are 3 amended JARS to copy!

    UPDATE2:

    After restarting the app on all 3 Applications Servers (forgot to do this have set ConsolidationMultiThreadScheme as a Global setting), I can get the Released patch to go as fast as the Dev patch (9.5 minutes is fastest run). I'm happy!

    I hope that those of you affected by this on 11.2.7/11.2.8 will soon get an equivalent patch.

    Steve

  • Thanos A
    Thanos A Consolidation System Manager Member Posts: 1,443 Silver Trophy

    Hi Steve,

    Exciting news after 3 years of complains with development.

    So, if I understand your results, you have managed to reduce the consolidation time from 60minutes to 9.5minutes? Have you tried to change the ConsolidationMultiThreadScheme=2 or more vCPU (We had also found that 12 was the magic number).

    Thank you,

    Thanos