12 Replies Latest reply: Feb 14, 2013 1:38 PM by user10757003 RSS

    FDM Data load

    990362
      When I try to load data for large files into FDM ,not able to load data.

      while loading data to FDM small file size data load is ok but larger file FDM load is not able to do.

      Please suggest on this

      Edited by: 987359 on Feb 12, 2013 8:53 AM
        • 1. Re: FDM Data load
          SH
          When you say large, how many records? Are you using the Bulk Insert method instead of SQL Insert (this is set on the location metadata screen)?
          • 2. Re: FDM Data load
            JTF85
            You may want to check if it is your file size that is the issue, or if it is a mapping issue, you can check this by using the Import Analysis reports that come in the default FDM reports.

            JTF
            • 3. Re: FDM Data load
              user10757003
              I have recently had the same problem, and is related to IIS settings llimiting the size of files when importing from a local directory. Unfortunately there is no FDM error message or entry in the log (under import text file) that relates to this.

              If you place the large file directly into the FDM Location inbox, then it should import ok.
              • 4. Re: FDM Data load
                SH
                You sould always try to load from the Inbox as it is much more performant as it usually resides on the FDM application server itself and therfore has no network overhead involved whereas the Upload option streams data across the network which can be very ineffivciennt for large amounts of data and subject to dropped network connections. I personally don't see why the Upload tab is even present in FDM (it casuses issues as outlined and offers no additional value that I can see) as you can browse to any file accessible on your network via the Select from Inbox tab and upload to the Inbox before loading.
                • 5. Re: FDM Data load
                  990362
                  I tried a small load approx. 200 records and the import/validate/export sections completed very quickly.
                  I tried a load half the size of the yesterdays file approx. 35000 rows and the load is hanging.

                  Are these processes actually doing anything
                  • 6. Re: FDM Data load
                    990362
                    I tried a small load approx. 200 records and the import/validate/export sections completed very quickly.
                    I tried a load half the size of the yesterdays file approx. 35000 rows and the load is hanging.

                    Are these processes actually doing anything
                    • 7. Re: FDM Data load
                      SH
                      Can you describe what you are doing?

                      1) Are you loading from the Inbox?
                      2) Do you have bulk insert selected for the location you are loading data to?
                      3) Do you have lots of conditional maps?
                      4) Are you using logic accounts?
                      • 8. Re: FDM Data load
                        990362
                        The actuals file I’ve been attempting to load is the smallest file we normally load and is approx. 1/10th of the largest files we ordinarily load overnight.
                        So data volume isn’t high at all, this should approx 15 -20 mins to load. These jobs have been running for over 20hrs, so I think there’s a problem.
                        • 9. Re: FDM Data load
                          SH
                          What about the answer o the above questions?
                          • 10. Re: FDM Data load
                            990362
                            Is there any problems in setting in hyperion when data loading from FDM essbase it still runs,not able to load for large file data
                            Please suggest
                            • 11. Re: FDM Data load
                              990362
                              I do not have bulk load not have many maps loading data as usual
                              • 12. Re: FDM Data load
                                user10757003
                                Have you checked the log option (under the analysis menu option) as this will allow you to check times for specific parts of the process.
                                e.g. Import Text File, Process Maps, total Import process, Intersection Validation, Adaptor export / load etc.

                                this might help to analyse where the issue is arising.

                                We import / load approx 500,000 records to HFM in about 20 minutes without intersection validation on a development applicaiton and this could be improved further by rewriting some of the mapping, so the volumes you are dealing with should be minutes. The same file imports / loads to Essbase in a couple of minutes with minimal mapping.

                                If it is Essbase you are loading, have you checked the clear script in the essbase adaptor Load script, (and cqalc script in the consolidaiton script) as this is generic and should be customised to your particular environment.