712: Unable to lock data store memory segment, error 12 while load new db

785849
    Hi Masters

    I am meeting the error "712: Unable to lock data store memory segment, error 12" while I want to load new database in memory.

    ttisql bk_rdb

    Copyright (c) 1996-2011, Oracle. All rights reserved.
    Type ? or "help" for help, type "exit" to quit ttIsql.

    connect "DSN=bk_rdb";
    712: Unable to lock data store memory segment, error 12
    The command failed.
    Done.

    Below is my tt's and os information. I just googled and make some adjustment in system control file, how it still can not work, can you please point me at right directions. thanks

    os :

    Linux igs_imdb02 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

    cpu:

    Intel(R) Xeon(R) CPU X5560 @ 2.80GHz x 7

    memory:

    free -g
    total used free shared buffers cached
    Mem: 15 10 5 0 0 0
    -/+ buffers/cache: 10 5
    Swap: 28 0 28


    cat /proc/meminfo
    MemTotal: 16411856 kB
    MemFree: 5340968 kB
    Buffers: 40736 kB
    Cached: 311236 kB
    SwapCached: 0 kB
    Active: 195552 kB
    Inactive: 295852 kB
    HighTotal: 0 kB
    HighFree: 0 kB
    LowTotal: 16411856 kB
    LowFree: 5340968 kB
    SwapTotal: 29989292 kB
    SwapFree: 29989292 kB
    Dirty: 28 kB
    Writeback: 0 kB
    AnonPages: 139468 kB
    Mapped: 28528 kB
    Slab: 43976 kB
    PageTables: 8212 kB
    NFS_Unstable: 0 kB
    Bounce: 0 kB
    CommitLimit: 32952340 kB
    Committed_AS: 587816 kB
    VmallocTotal: 34359738367 kB
    VmallocUsed: 278164 kB
    VmallocChunk: 34359459099 kB
    HugePages_Total: 5120
    HugePages_Free: 5120
    HugePages_Rsvd: 0
    Hugepagesize: 2048 kB

    /etc/security/limits.conf have below entries

    * soft memlock 23545248
    * hard memlock 23545248

    DSN attributes

    [BK_RDB]
    Driver=/ora/TimesTen/tt1121/lib/libtten.so
    DataStore=/ora/TimesTen/tt1121/dbs
    DatabaseCharacterSet=US7ASCII
    TempSize=1024
    PermSize=12783
    MemoryLock=4

    ttversion
    TimesTen Release 11.2.1.8.0 (64 bit Linux/x86_64) (tt1121:53388) 2011-02-02T02:20:46Z
    Instance admin: tt
    Instance home directory: /ora/TimesTen/tt1121
    Group owner: ttadmin
    Daemon home directory: /ora/TimesTen/tt1121/info
    PL/SQL enabled.

    Edited by: user1997872 on Apr 12, 2011 2:47 AM
      • 1. Re: 712: Unable to lock data store memory segment, error 12 while load new db
        785849
        I want MemoryLock=4 for enhance performance and setup /etc/security/limits.conf for memlock's values

        I saw below url and made adjustment accordingly

        Re: failed to lock memory

        but it still run into error.

        Edited by: user1997872 on Apr 12, 2011 2:36 AM
        • 2. Re: 712: Unable to lock data store memory segment, error 12 while load new db
          Jspalmer-Oracle
          Your Huge Pages (HugePages_Total) looks too low

          5120 * 2048 = 10485760KB ie 10737418240 bytes or 10GB. Your PermSize on its own is 12GB
          • 3. Re: 712: Unable to lock data store memory segment, error 12 while load new db
            Jspalmer-Oracle
            actually i should qualify that - do you have "-linuxLargePageAlignment +nnnn+" set in your ttendaemon.options file?
            • 4. Re: 712: Unable to lock data store memory segment, error 12 while load new db
              ChrisJenkins-Oracle
              errno 12 = ENOMEM. There are a few possible causes.

              If you are trying to use Huge Pages (i.e. you have specified '-linuxLargePageAlignment 2' in your ttendaemon.options file and have restarted the daemon) then you have only allocated 10 GB of huge pages while your datastore will need a little over 13 GB of huge pages. Since your system only has 16 GB of memory I would be very cautios about trying to use huge pages for this datastore; there will be very little non-huge page memory left for reuglar usage.

              If you are not trying to use huge pages then your values for memlock look okay though if you have recently edited them then you need to stop the TT demon, log out, log back in as TT instance administrator and then restart the daemon (since these limits are only set at login time).

              If you still get this error then it could be that you just don't have enough memory... The info from cat /proc/meminfo shows only a little over 5 GB free and you need over 13 Gb to load this datatsore into memory.

              Chris
              • 5. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                785849
                thanks Chris and Jspalmer

                I have specified '-linuxLargePageAlignment 2' in my ttendaemon.options file and have restarted the daemon

                and also increased the value hugesize in /proc/sys/vm/nr_hugepages and beboot the box. not changed of memlock.

                however I still get same error. How can I get right operations

                there only stand long tt database on this box, no others applications so I want tt database use the physical memory at maximum


                cat /proc/meminfo
                MemTotal: 16411856 kB
                MemFree: 1360708 kB
                Buffers: 40844 kB
                Cached: 313512 kB
                SwapCached: 0 kB
                Active: 196524 kB
                Inactive: 297092 kB
                HighTotal: 0 kB
                HighFree: 0 kB
                LowTotal: 16411856 kB
                LowFree: 1360708 kB
                SwapTotal: 29989292 kB
                SwapFree: 29989292 kB
                Dirty: 8 kB
                Writeback: 0 kB
                AnonPages: 139248 kB
                Mapped: 28348 kB
                Slab: 44700 kB
                PageTables: 8112 kB
                NFS_Unstable: 0 kB
                Bounce: 0 kB
                CommitLimit: 30965780 kB
                Committed_AS: 589720 kB
                VmallocTotal: 34359738367 kB
                VmallocUsed: 278164 kB
                VmallocChunk: 34359459099 kB
                HugePages_Total: 7060
                HugePages_Free: 7060
                HugePages_Rsvd: 0
                Hugepagesize: 2048 kB
                • 6. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                  Jspalmer-Oracle
                  As user tt / ttadmin can you run ulimit -l to doublecheck what that user's memlock limit is? Otherwise, the huge page settings should be okay for your datastore as sized in your earlier entry. You could try rebooting the box to see if that's what it needs. You've set memlock to be ~24GB though the box itself only has a MemTotal of ~16GB. What happens if you set memlock to be less than MemTotal ? I see Note 361468.1 recommends it should be less than the total RAM. Also - could you try allocating a much smaller datastore - say 100MB - and verify that Huge Pages are actually used for its memory segment (verify that HugePages_Free reduces).

                  Otherwise are you sure it's using those DSN settings, and not using - larger - Perm and Temp size settings from another sys.odbc.ini? ttIsql when you connect will show you the DSN values being used.
                  • 7. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                    785849
                    jspalmer, thanks for you reply.

                    I made changes of memlock for user limit as below and reboot box.
                    and I make sure I use the correct DSN source file.

                    however I am still getting same error.
                    ulimit -l
                    15172052

                    Now I just removed the Memlock=4 from DSN attributes and open the database successfully

                    ttisql bk_rdb

                    Copyright (c) 1996-2011, Oracle. All rights reserved.
                    Type ? or "help" for help, type "exit" to quit ttIsql.

                    connect "DSN=bk_rdb";
                    Connection successful: DSN=BK_RDB;UID=tt;DataStore=/ora/TimesTen/tt1121/dbs;DatabaseCharacterSet=US7ASCII;ConnectionCharacterSet=US7ASCII;DRIVER=/ora/TimesTen/tt1121/lib/libtten.so;PermSize=13362;TempSize=1024;TypeMode=0;
                    (Default setting AutoCommit=1)

                    now, my /proc/meminfo

                    MemTotal: 16411856 kB
                    MemFree: 984564 kB
                    Buffers: 43056 kB
                    Cached: 649076 kB
                    SwapCached: 0 kB
                    Active: 283412 kB
                    Inactive: 579252 kB
                    HighTotal: 0 kB
                    HighFree: 0 kB
                    LowTotal: 16411856 kB
                    LowFree: 984564 kB
                    SwapTotal: 29989292 kB
                    SwapFree: 29989292 kB
                    Dirty: 8 kB
                    Writeback: 0 kB
                    AnonPages: 170568 kB
                    Mapped: 97904 kB
                    Slab: 51840 kB
                    PageTables: 8876 kB
                    NFS_Unstable: 0 kB
                    Bounce: 0 kB
                    CommitLimit: 30965780 kB
                    Committed_AS: 15529996 kB
                    VmallocTotal: 34359738367 kB
                    VmallocUsed: 278164 kB
                    VmallocChunk: 34359459099 kB
                    HugePages_Total: 7060
                    HugePages_Free: 7060
                    HugePages_Rsvd: 0
                    Hugepagesize: 2048 kB

                    so i am not sure if my tt used huge pages and why I can not use the memlock.
                    • 8. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                      ChrisJenkins-Oracle
                      It looks very much like large pages are not being used. Did you capture that /proc/meminfo while the datastore was loaded in memory?

                      In order for TimesTen to be able to use huge pages then, as well as setting the option in the ttendaemon.options file, the following kernel parameter must also be set correctly:

                      vm.hugetlb_shm_group = 1000

                      The value (1000 in my example) is the numeric group id (from /etc/group) of a group which the TimesTen instance administrator user belongs to. Here is an example from one of my systems:

                      [cdjenkin@reliant1 ~]$ ttversion
                      TimesTen Release 11.2.1.8.0 (64 bit Linux/x86_64) (tt1121:53388) 2011-02-02T02:20:46Z
                      Instance admin: cdjenkin
                      Instance home directory: /timesten/TimesTen/tt1121
                      Group owner: ttuser
                      Daemon home directory: /timesten/TimesTen/tt1121/info
                      PL/SQL enabled.
                      [cdjenkin@reliant1 ~]$ id
                      uid=*500(cdjenkin)* gid=501(cdjenkin) groups=501(cdjenkin),504(dba),505(oinstall),*1000(timesten)*,1100(ttuser)
                      [cdjenkin@reliant1 ~]$

                      Without this setting the daemon will not be allowed to use huge pages and this could also be the reason why the memlock is being denied.

                      Chris
                      • 9. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                        785849
                        Hi Chris,

                        yes, the above information of /proc/meminfo I posted was captured while the data store load into memory. it was looks hugepage not used

                        I just did configure the group id as you mentioned base on description in install guide.

                        but I am not sure why the hugpage was not used.

                        anyway, I was working on setup imdb cache infrastructure and have already setup another node as same configurations as this one, and the hugepages might be used as below

                        cat /proc/meminfo
                        MemTotal: 4930908 kB
                        MemFree: 164820 kB
                        Buffers: 181004 kB
                        Cached: 1981516 kB
                        SwapCached: 4 kB
                        Active: 1602120 kB
                        Inactive: 883704 kB
                        HighTotal: 0 kB
                        HighFree: 0 kB
                        LowTotal: 4930908 kB
                        LowFree: 164820 kB
                        SwapTotal: 6979576 kB
                        SwapFree: 6979572 kB
                        Dirty: 20 kB
                        Writeback: 0 kB
                        AnonPages: 323360 kB
                        Mapped: 499720 kB
                        Slab: 116640 kB
                        PageTables: 36908 kB
                        NFS_Unstable: 0 kB
                        Bounce: 0 kB
                        CommitLimit: 8396452 kB
                        Committed_AS: 5098428 kB
                        VmallocTotal: 34359738367 kB
                        VmallocUsed: 266852 kB
                        VmallocChunk: 34359470079 kB
                        HugePages_Total: 1024
                        HugePages_Free: 1008
                        HugePages_Rsvd: 0
                        Hugepagesize: 2048 kB

                        so I still not get clew for fixing this issue, however I have to going to next for complete the whole cache infrastructure setup.

                        now, I am meeting another issue for cache grip attach, and send new post as below. Can you please help on that again. thanks

                        getting error while attach cache grid: 3344: Error creating grid threads:

                        Edited by: user1997872 on Apr 18, 2011 11:57 PM
                        • 10. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                          ChrisJenkins-Oracle
                          Unless you now have a very small datastore then TimesTen isn't using hugepages here either:

                          HugePages_Total: 1024
                          HugePages_Free: 1008

                          I suggest that you check all aspects of the config (including the kernel parameter I mentioned earlier, the memlock value in limits.conf, the -linuxLargePageAlignment flag in the ttendaemon.options file and then reboot the machine and see if huge pages work then. I stringly suspetrc that some aspect of your config is wrong.

                          Chris
                          • 11. Re: 712: Unable to lock data store memory segment, error 12 while load new db
                            785849
                            OK, Chris, I will do test again base on your suggestions.

                            will let you know the results.

                            thanks