6 Replies Latest reply: Apr 7, 2012 8:35 AM by 926170 RSS

    dsadm import become slow , help me

    926170
      I use dsadm import to load ldif data to ldap.

      At beginning the speed is OK , every 20 seconds about 80 thousands entries loaded.
      [06/Apr/2012:14:24:03 +0800] - import wasu: Processed 83921 entries -- average rate 4196.1/sec, recent rate 4196.1/sec, hit ratio 0% -- written to database.
      [06/Apr/2012:14:24:23 +0800] - import wasu: Processed 170686 entries -- average rate 4267.2/sec, recent rate 4267.1/sec, hit ratio 100% -- written to database.
      [06/Apr/2012:14:24:43 +0800] - import wasu: Processed 252173 entries -- average rate 4202.9/sec, recent rate 4206.3/sec, hit ratio 100% -- written to database.
      [06/Apr/2012:14:25:03 +0800] - import wasu: Processed 335231 entries -- average rate 4190.4/sec, recent rate 4113.6/sec, hit ratio 100% -- written to database.

      But later the speed become slow
      [06/Apr/2012:16:01:17 +0800] - import wasu: Processed 11042450 entries (pass 3) -- average rate 180.7/sec, recent rate 243.7/sec, hit ratio 96% -- written to database.
      [06/Apr/2012:16:01:37 +0800] - import wasu: Processed 11047608 entries (pass 3) -- average rate 181.7/sec, recent rate 246.1/sec, hit ratio 96% -- written to database.
      [06/Apr/2012:16:01:57 +0800] - import wasu: Processed 11051250 entries (pass 3) -- average rate 181.7/sec, recent rate 220.0/sec, hit ratio 96% -- written to database.

      My ldif file is about 3.7G , and have 10 millions entries.

      How can I increase the import speed

      Thanks
        • 1. Re: dsadm import become slow , help me
          Marco Milo-Oracle
          Sorry for starting asking questions instead of providing answers, but the informations provided here are not complete...

          HW-Related:
          - Which is your physical HW platform? are you using any virtualization?
          - How busy is the CPU, I/O subsystem and RAM?
          DS-Related:
          - Which version are you using?
          - How did you sized your import cache?
          - How many indexes are there?
          - Do you have any special setting for the allids threshold?
          - How big is the average entry size?
          - Do entries contain a lot of replication metadata?
          • 2. Re: dsadm import become slow , help me
            926170
            HW-Related:
            1 . My HW platform , dell 2950 4G memory , Intel(R) Xeon(R) CPU 5120 @ 1.86GHz *4
            2 . When importing , CPU is about 40%

            DS-Related:
            1 . I use ODSEE 11.1.1.3
            2 . I use default value for all cache . I guess import cache is 64M
            3 . I didn't create any index when importing
            4 . I didn't have any special setting for the allids threshold
            5 . Entries do not have any replication metadata

            Thanks for reply
            • 3. Re: dsadm import become slow , help me
              Marco Milo-Oracle
              Well, leaving just the 'default' values may lead to performance issues and bottlenecks since the tuning of a platform for optimal performances is specifically tied to that environment... but anyway, it's a starting point.

              You said you're running 11.1.1.3, which is good enough, but actually 11.1.1.5 is out already, so if this it's an initial deployment and you're not bound to a specific version go for the latest release...

              Regarding your performance issue, you may have some benefit increasing the import cache up to 2 GB and perform an off-line initialization; relevant official product documentation can be found at:

              http://docs.oracle.com/cd/E19656-01/821-1507/fnyty/index.html

              http://docs.oracle.com/cd/E19656-01/821-1509/import-cache-size-5dsconf/index.html

              I don't know what else is running on the system, but with only 4GB of RAM, you may hit some memory limitation... it's crucial that your system doesn't start swapping since this paging activity will almost certainly kill your server.

              Last but not least, I forgot to ask you which is the O.S.? 32 or 64-bits? did you manage to measure somehow the I/O 'performances' ?
              • 4. Re: dsadm import become slow , help me
                926170
                Thanks Marco

                My OS is 64-bits , Red Hat Enterprise Linux Server release 5.7.

                My question

                1 . dsadm import is an off-line initialization . When ns-slapd down , import cache will be used ?

                2 . dsconf import can also initialization . Which command should I use , dsadm import or dsconf import

                Thanks
                • 5. Re: dsadm import become slow , help me
                  Marco Milo-Oracle
                  Yes, dsadm import performs the off-line initialization using the import cache (and it is performed on a 'local' off-line instance), whereas you can use the dsconf import command on an already running Directory Server instance (local or remote); again the relevant official product documentation is:

                  http://docs.oracle.com/cd/E19656-01/821-1504/ganvk/index.html

                  HTH,
                  marco
                  • 6. Re: dsadm import become slow , help me
                    926170
                    I set import cache for 2G , and test import again . The effect seems not very obviously . It takes 4.5 hours.

                    The import process split for 2 pass .

                    pass 1 speed is OK , but it also become slow
                    ## Processed 10563879 entries -- average rate 3008.8/sec, recent rate 1570.0/sec, hit ratio 100% -- written to database.

                    pass 2 speed is too slow
                    ## Decided to end this pass because the progress rate has dropped below the 60% threshold for too long.
                    ## Ending pass number 1 ...
                    ## Foreman is done; waiting for workers to finish...
                    ## Processed 10772794 entries -- average rate 2886.6/sec, recent rate 146.9/sec, hit ratio 97% -- written to database.
                    ## Workers finished; cleaning up...
                    ## Workers cleaned up.
                    ## Sweeping files for merging later...
                    ## Sweep done.
                    ## Beginning pass number 2
                    ## Processed 10776101 entries (pass 2) -- average rate 165.3/sec, recent rate 165.3/sec, hit ratio 0% -- written to database.
                    ## Processed 10778791 entries (pass 2) -- average rate 149.9/sec, recent rate 149.9/sec, hit ratio 97% -- written to database.
                    ## Processed 10781874 entries (pass 2) -- average rate 151.3/sec, recent rate 144.3/sec, hit ratio 97% -- written to database.

                    Except import cache , is there any other ideas to increase the import speed
                    Thanks