11 Replies Latest reply on Apr 20, 2011 1:50 AM by 785849

    712: Unable to lock data store memory segment, error 12 while load new db

    785849
      Hi Masters

      I am meeting the error "712: Unable to lock data store memory segment, error 12" while I want to load new database in memory.

      ttisql bk_rdb

      Copyright (c) 1996-2011, Oracle. All rights reserved.
      Type ? or "help" for help, type "exit" to quit ttIsql.

      connect "DSN=bk_rdb";
      712: Unable to lock data store memory segment, error 12
      The command failed.
      Done.

      Below is my tt's and os information. I just googled and make some adjustment in system control file, how it still can not work, can you please point me at right directions. thanks

      os :

      Linux igs_imdb02 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

      cpu:

      Intel(R) Xeon(R) CPU X5560 @ 2.80GHz x 7

      memory:

      free -g
      total used free shared buffers cached
      Mem: 15 10 5 0 0 0
      -/+ buffers/cache: 10 5
      Swap: 28 0 28


      cat /proc/meminfo
      MemTotal: 16411856 kB
      MemFree: 5340968 kB
      Buffers: 40736 kB
      Cached: 311236 kB
      SwapCached: 0 kB
      Active: 195552 kB
      Inactive: 295852 kB
      HighTotal: 0 kB
      HighFree: 0 kB
      LowTotal: 16411856 kB
      LowFree: 5340968 kB
      SwapTotal: 29989292 kB
      SwapFree: 29989292 kB
      Dirty: 28 kB
      Writeback: 0 kB
      AnonPages: 139468 kB
      Mapped: 28528 kB
      Slab: 43976 kB
      PageTables: 8212 kB
      NFS_Unstable: 0 kB
      Bounce: 0 kB
      CommitLimit: 32952340 kB
      Committed_AS: 587816 kB
      VmallocTotal: 34359738367 kB
      VmallocUsed: 278164 kB
      VmallocChunk: 34359459099 kB
      HugePages_Total: 5120
      HugePages_Free: 5120
      HugePages_Rsvd: 0
      Hugepagesize: 2048 kB

      /etc/security/limits.conf have below entries

      * soft memlock 23545248
      * hard memlock 23545248

      DSN attributes

      [BK_RDB]
      Driver=/ora/TimesTen/tt1121/lib/libtten.so
      DataStore=/ora/TimesTen/tt1121/dbs
      DatabaseCharacterSet=US7ASCII
      TempSize=1024
      PermSize=12783
      MemoryLock=4

      ttversion
      TimesTen Release 11.2.1.8.0 (64 bit Linux/x86_64) (tt1121:53388) 2011-02-02T02:20:46Z
      Instance admin: tt
      Instance home directory: /ora/TimesTen/tt1121
      Group owner: ttadmin
      Daemon home directory: /ora/TimesTen/tt1121/info
      PL/SQL enabled.

      Edited by: user1997872 on Apr 12, 2011 2:47 AM
        • 1. Re: 712: Unable to lock data store memory segment, error 12 while load new db
          785849
          I want MemoryLock=4 for enhance performance and setup /etc/security/limits.conf for memlock's values

          I saw below url and made adjustment accordingly

          Re: failed to lock memory

          but it still run into error.

          Edited by: user1997872 on Apr 12, 2011 2:36 AM
          • 2. Re: 712: Unable to lock data store memory segment, error 12 while load new db
            Jspalmer-Oracle
            Your Huge Pages (HugePages_Total) looks too low

            5120 * 2048 = 10485760KB ie 10737418240 bytes or 10GB. Your PermSize on its own is 12GB
            • 3. Re: 712: Unable to lock data store memory segment, error 12 while load new db
              Jspalmer-Oracle
              actually i should qualify that - do you have "-linuxLargePageAlignment +nnnn+" set in your ttendaemon.options file?
              • 4. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                ChrisJenkins-Oracle
                errno 12 = ENOMEM. There are a few possible causes.

                If you are trying to use Huge Pages (i.e. you have specified '-linuxLargePageAlignment 2' in your ttendaemon.options file and have restarted the daemon) then you have only allocated 10 GB of huge pages while your datastore will need a little over 13 GB of huge pages. Since your system only has 16 GB of memory I would be very cautios about trying to use huge pages for this datastore; there will be very little non-huge page memory left for reuglar usage.

                If you are not trying to use huge pages then your values for memlock look okay though if you have recently edited them then you need to stop the TT demon, log out, log back in as TT instance administrator and then restart the daemon (since these limits are only set at login time).

                If you still get this error then it could be that you just don't have enough memory... The info from cat /proc/meminfo shows only a little over 5 GB free and you need over 13 Gb to load this datatsore into memory.

                Chris
                • 5. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                  785849
                  thanks Chris and Jspalmer

                  I have specified '-linuxLargePageAlignment 2' in my ttendaemon.options file and have restarted the daemon

                  and also increased the value hugesize in /proc/sys/vm/nr_hugepages and beboot the box. not changed of memlock.

                  however I still get same error. How can I get right operations

                  there only stand long tt database on this box, no others applications so I want tt database use the physical memory at maximum


                  cat /proc/meminfo
                  MemTotal: 16411856 kB
                  MemFree: 1360708 kB
                  Buffers: 40844 kB
                  Cached: 313512 kB
                  SwapCached: 0 kB
                  Active: 196524 kB
                  Inactive: 297092 kB
                  HighTotal: 0 kB
                  HighFree: 0 kB
                  LowTotal: 16411856 kB
                  LowFree: 1360708 kB
                  SwapTotal: 29989292 kB
                  SwapFree: 29989292 kB
                  Dirty: 8 kB
                  Writeback: 0 kB
                  AnonPages: 139248 kB
                  Mapped: 28348 kB
                  Slab: 44700 kB
                  PageTables: 8112 kB
                  NFS_Unstable: 0 kB
                  Bounce: 0 kB
                  CommitLimit: 30965780 kB
                  Committed_AS: 589720 kB
                  VmallocTotal: 34359738367 kB
                  VmallocUsed: 278164 kB
                  VmallocChunk: 34359459099 kB
                  HugePages_Total: 7060
                  HugePages_Free: 7060
                  HugePages_Rsvd: 0
                  Hugepagesize: 2048 kB
                  • 6. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                    Jspalmer-Oracle
                    As user tt / ttadmin can you run ulimit -l to doublecheck what that user's memlock limit is? Otherwise, the huge page settings should be okay for your datastore as sized in your earlier entry. You could try rebooting the box to see if that's what it needs. You've set memlock to be ~24GB though the box itself only has a MemTotal of ~16GB. What happens if you set memlock to be less than MemTotal ? I see Note 361468.1 recommends it should be less than the total RAM. Also - could you try allocating a much smaller datastore - say 100MB - and verify that Huge Pages are actually used for its memory segment (verify that HugePages_Free reduces).

                    Otherwise are you sure it's using those DSN settings, and not using - larger - Perm and Temp size settings from another sys.odbc.ini? ttIsql when you connect will show you the DSN values being used.
                    • 7. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                      785849
                      jspalmer, thanks for you reply.

                      I made changes of memlock for user limit as below and reboot box.
                      and I make sure I use the correct DSN source file.

                      however I am still getting same error.
                      ulimit -l
                      15172052

                      Now I just removed the Memlock=4 from DSN attributes and open the database successfully

                      ttisql bk_rdb

                      Copyright (c) 1996-2011, Oracle. All rights reserved.
                      Type ? or "help" for help, type "exit" to quit ttIsql.

                      connect "DSN=bk_rdb";
                      Connection successful: DSN=BK_RDB;UID=tt;DataStore=/ora/TimesTen/tt1121/dbs;DatabaseCharacterSet=US7ASCII;ConnectionCharacterSet=US7ASCII;DRIVER=/ora/TimesTen/tt1121/lib/libtten.so;PermSize=13362;TempSize=1024;TypeMode=0;
                      (Default setting AutoCommit=1)

                      now, my /proc/meminfo

                      MemTotal: 16411856 kB
                      MemFree: 984564 kB
                      Buffers: 43056 kB
                      Cached: 649076 kB
                      SwapCached: 0 kB
                      Active: 283412 kB
                      Inactive: 579252 kB
                      HighTotal: 0 kB
                      HighFree: 0 kB
                      LowTotal: 16411856 kB
                      LowFree: 984564 kB
                      SwapTotal: 29989292 kB
                      SwapFree: 29989292 kB
                      Dirty: 8 kB
                      Writeback: 0 kB
                      AnonPages: 170568 kB
                      Mapped: 97904 kB
                      Slab: 51840 kB
                      PageTables: 8876 kB
                      NFS_Unstable: 0 kB
                      Bounce: 0 kB
                      CommitLimit: 30965780 kB
                      Committed_AS: 15529996 kB
                      VmallocTotal: 34359738367 kB
                      VmallocUsed: 278164 kB
                      VmallocChunk: 34359459099 kB
                      HugePages_Total: 7060
                      HugePages_Free: 7060
                      HugePages_Rsvd: 0
                      Hugepagesize: 2048 kB

                      so i am not sure if my tt used huge pages and why I can not use the memlock.
                      • 8. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                        ChrisJenkins-Oracle
                        It looks very much like large pages are not being used. Did you capture that /proc/meminfo while the datastore was loaded in memory?

                        In order for TimesTen to be able to use huge pages then, as well as setting the option in the ttendaemon.options file, the following kernel parameter must also be set correctly:

                        vm.hugetlb_shm_group = 1000

                        The value (1000 in my example) is the numeric group id (from /etc/group) of a group which the TimesTen instance administrator user belongs to. Here is an example from one of my systems:

                        [cdjenkin@reliant1 ~]$ ttversion
                        TimesTen Release 11.2.1.8.0 (64 bit Linux/x86_64) (tt1121:53388) 2011-02-02T02:20:46Z
                        Instance admin: cdjenkin
                        Instance home directory: /timesten/TimesTen/tt1121
                        Group owner: ttuser
                        Daemon home directory: /timesten/TimesTen/tt1121/info
                        PL/SQL enabled.
                        [cdjenkin@reliant1 ~]$ id
                        uid=*500(cdjenkin)* gid=501(cdjenkin) groups=501(cdjenkin),504(dba),505(oinstall),*1000(timesten)*,1100(ttuser)
                        [cdjenkin@reliant1 ~]$

                        Without this setting the daemon will not be allowed to use huge pages and this could also be the reason why the memlock is being denied.

                        Chris
                        • 9. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                          785849
                          Hi Chris,

                          yes, the above information of /proc/meminfo I posted was captured while the data store load into memory. it was looks hugepage not used

                          I just did configure the group id as you mentioned base on description in install guide.

                          but I am not sure why the hugpage was not used.

                          anyway, I was working on setup imdb cache infrastructure and have already setup another node as same configurations as this one, and the hugepages might be used as below

                          cat /proc/meminfo
                          MemTotal: 4930908 kB
                          MemFree: 164820 kB
                          Buffers: 181004 kB
                          Cached: 1981516 kB
                          SwapCached: 4 kB
                          Active: 1602120 kB
                          Inactive: 883704 kB
                          HighTotal: 0 kB
                          HighFree: 0 kB
                          LowTotal: 4930908 kB
                          LowFree: 164820 kB
                          SwapTotal: 6979576 kB
                          SwapFree: 6979572 kB
                          Dirty: 20 kB
                          Writeback: 0 kB
                          AnonPages: 323360 kB
                          Mapped: 499720 kB
                          Slab: 116640 kB
                          PageTables: 36908 kB
                          NFS_Unstable: 0 kB
                          Bounce: 0 kB
                          CommitLimit: 8396452 kB
                          Committed_AS: 5098428 kB
                          VmallocTotal: 34359738367 kB
                          VmallocUsed: 266852 kB
                          VmallocChunk: 34359470079 kB
                          HugePages_Total: 1024
                          HugePages_Free: 1008
                          HugePages_Rsvd: 0
                          Hugepagesize: 2048 kB

                          so I still not get clew for fixing this issue, however I have to going to next for complete the whole cache infrastructure setup.

                          now, I am meeting another issue for cache grip attach, and send new post as below. Can you please help on that again. thanks

                          getting error while attach cache grid: 3344: Error creating grid threads:

                          Edited by: user1997872 on Apr 18, 2011 11:57 PM
                          • 10. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                            ChrisJenkins-Oracle
                            Unless you now have a very small datastore then TimesTen isn't using hugepages here either:

                            HugePages_Total: 1024
                            HugePages_Free: 1008

                            I suggest that you check all aspects of the config (including the kernel parameter I mentioned earlier, the memlock value in limits.conf, the -linuxLargePageAlignment flag in the ttendaemon.options file and then reboot the machine and see if huge pages work then. I stringly suspetrc that some aspect of your config is wrong.

                            Chris
                            • 11. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                              785849
                              OK, Chris, I will do test again base on your suggestions.

                              will let you know the results.

                              thanks