8 Replies Latest reply: Nov 5, 2012 4:27 AM by User10403489-Oracle RSS

    Performance issue with FDM when importing data

    User10403489-Oracle
      In the FDM Web console, a performance issue has been detected when importing data (.txt)

      In less than 10 seconds the ".txt" and the ".log" files are created the INBOX folder (the ".txt" file) and in the OUTBOX\Logs (the ".log" file).

      At that moment, system shows the message "Processing, please wait” during 10 minutes. Eventually the information is displayed, however if we want to see the second page, we have to wait more than 20 seconds.

      It seems a performance issue when system tries to show the imported data in the web page.

      It has been also noted that when a user tries to import a txt file directly clicking on the tab "Select File From Inbox", the user has to also wait other 10 minutes before the information is displayed on the web page.


      Thx in advance!

      Cheers

      Matteo
        • 1. Re: Performance issue with FDM when importing data
          Stuart Game
          Hi Matteo

          How much data is being imported / displayed when users are interacting with the system.

          There is a report that may help you to analyse this but unfortunately I cannot remember what it is called and don't have access to a system to check. I do remember that it breaks down the import process into stages showing how long it takes to process each mapping step and the overall time.

          I suspect that what you are seeing is normal behaviour but that isn't to say that performance improvements are not possible.

          The copying of files is the first part of the import process before FDM then starts the import so that will be quick. The processing is then the time taken to import the records, process the mapping and write to the tables. If users are clicking 'Select file from Inbox' then they are re-importing so it will take just as long as it would for you to import it, they are not just asking to retrieve previously imported data.

          Hope this helps
          Stuart
          • 2. Re: Performance issue with FDM when importing data
            JTF85
            Have you run the reports for Import Analysis? By default this is in the System Reports folder. The report will help to show you if it is an issue of the actual file transfer, the mapping process, or if it is something else. As you may be aware the mapping logic is applied during the import step, so a mapping logic issue could cause problems in this area.

            JTF
            • 3. Re: Performance issue with FDM when importing data
              User10403489-Oracle
              Thanks,

              Your help is useful, I suppose the trouble is with the import formats, but I must test on the customer environment running the import performance analysis.
              With this report I can focus the problem.


              Thanks a lot!!!!!!!


              Matteo
              • 4. Re: Performance issue with FDM when importing data
                User10403489-Oracle
                I Gurus,

                So I lost a lot of minutes to import process map, Memo Reassign Time and File Archive Time.

                Process Description Duration I/O Source Event Information Date Start Time End Time
                Import: Process Map 152 Wildcards For BK_INDIVIDUAL Flujos y Detalles Rules Processed - (5) 09/05/2012 8:19:36 AM 8:22:08 AM
                Period: Ago - 2012, Category: 12

                Import: Process Map 169 Wildcards For BK_INDIVIDUAL Fondo Comercio Rules Processed - (1) 09/05/2012 8:22:08 AM 8:24:57 AM
                Period: Ago - 2012, Category: 12

                File Archive Time 2418

                Memo Reassign Time 0 Invalid Grouping: Total does not foot

                Sometimes at the end of the report there is that red error: "Invalid Grouping Total does not foot" for the Process Description : Memo Reassign Time.

                Any helps?

                Trouble should be the import format?

                Thanks

                Cheers

                Matteo
                • 5. Re: Performance issue with FDM when importing data
                  User10403489-Oracle
                  Hi Gurus,

                  So I lost a lot of minutes to import process map, Memo Reassign Time and File Archive Time.

                  Process Description Duration I/O Source Event Information Date Start Time End Time
                  Import: Process Map 152 Wildcards For BK_INDIVIDUAL Flujos y Detalles Rules Processed - (5) 09/05/2012 8:19:36 AM 8:22:08 AM
                  Period: Ago - 2012, Category: 12

                  Import: Process Map 169 Wildcards For BK_INDIVIDUAL Fondo Comercio Rules Processed - (1) 09/05/2012 8:22:08 AM 8:24:57 AM
                  Period: Ago - 2012, Category: 12

                  File Archive Time 2418

                  Memo Reassign Time 0 Invalid Grouping: Total does not foot

                  Sometimes at the end of the report there is that red error: "Invalid Grouping Total does not foot" for the Process Description: Memo Reassign Time.

                  Any helps?

                  The trouble should be the import format?

                  Thx

                  Matteo
                  • 6. Re: Performance issue with FDM when importing data
                    JeffJon
                    I would focus on the Wildcard maps. Can you move any of your wildcards to Explcit Mappings? The Wildcards will always take the most processing time compared to the other mapping types available.
                    • 7. Re: Performance issue with FDM when importing data
                      User10403489-Oracle
                      Recalculation of Logic Accounts and Maps should be helpful?

                      I don't know if is possible move wildcards to Explicit Mappings, I have study the different between wildcards and Explicit Mappings.

                      Thanks for help me!!!

                      Matteo
                      • 8. Re: Performance issue with FDM when importing data
                        User10403489-Oracle
                        Hi,

                        The report shows that FDM is using woldcard for this POV but I can see on Activities > Maps (For the correct POV) that the Mapping is Explicit.

                        Any suggections?

                        Thx

                        Cheers

                        Matteo