9 Replies Latest reply on Aug 6, 2013 11:22 AM by ChrisJenkins-Oracle

    Max locked memory


      Hi Chris,


      1. Following information about memory lock is from TimesTen documentation:



      On Solaris, Linux and Windows 64-bit systems, TimesTen enables applications that connect to a shared database to specify whether the real memory should be locked while the database is being loaded into memory or while the store is in memory. If the physical memory used for the database is locked, the operating system's virtual memory subsystem cannot borrow that memory for other uses. No part of the database is ever paged out but this could lead to memory shortages in a system that is under configured with RAM. While memory locking can improve database load performance, it may impede other applications on the same computer.


      On Linux systems, set the groupname in the MemLock setting to be the same as the instance administrator in the /etc/security/limits.conf file. Set the value of MemLock to be at least as large as the TimesTen database shared memory segment.



      2. Following is our persize and tempsize for two datastores (a and b) present in the system

      a. PermSize=8000



      b. PermSize=8000



      3. We have set the TimesTen attribute MemoryLock=4 for our datastores.


      4. My "max locked memory" is 64k

      #ulimit -l



      5. The system has 24GB of RAM.


      6. ipcs

      ------ Shared Memory Segments --------

      key        shmid      owner      perms      bytes      nattch     status

      0x5d0500ac 0          root       660        1048576    24

      0x11040012 32769      root       660        8788663088 29                locked

      0x12040012 65538      root       660        268435456  29

      0x11040013 98307      root       660        9199704648 14                locked

      0x12040013 131076     root       660        268435456  14





      As TimesTen suggests "Set the value of MemLock to be at least as large as the TimesTen database shared memory segment". What is its significance?

      What we will be lacking if we retain the "max locked memory" value to 64 which is default value for Linux?




        • 1. Re: Max locked memory

          --- Continuation to the above posting:


          Following is the information of shared memory configured i my system.


          sysctl -a |grep shm

          kernel.shmmax = 17989369856

          kernel.shmall = 6008146





          • 2. Re: Max locked memory

            Hi Pratheej,


            It's a best practice recommendation to use MemoryLock=4 in you odbc.ini setting for your DSN(s). The reason for this is as described in the docs so that the TT shared memory segment is not a candidate to be swapped out of memory.


            For this to happen the OS has to be configured to accommodate this locked shared memory segment. One of these OS parameters is memlock. If memlock is not large enough to accommodate the whole of the TT shared memory segment then the TT shared memory segment won't be able to be locked.


            Your step 4 above is a little confusing. It looks like that is for the user root? What is the memlock value for the instance administrator, the user who starts and stops the TT daemon? I suspect it's a lot bigger as your output in step 6 implies there are 2 locked shared memory segments of about 8GB each which could be your TT databases? So it might be that your system is already configured to lock the TT segment?



            • 3. Re: Max locked memory

              Hello Tim,

              Thanks a lot for your reply.



              We are using 'root' user as the instance administrator and following is the ulimit -a output.





              [12:02:57 root@S2PXDM]$

              ulimit -a

              core file size          (blocks, -c) unlimited

              data seg size           (kbytes, -d) unlimited

              scheduling priority             (-e) 0

              file size               (blocks, -f) unlimited

              pending signals                 (-i) 187095

              max locked memory       (kbytes, -l) 64

              max memory size         (kbytes, -m) unlimited

              open files                      (-n) 1024

              pipe size            (512 bytes, -p) 8

              POSIX message queues     (bytes, -q) 819200

              real-time priority              (-r) 0

              stack size              (kbytes, -s) 2048

              cpu time               (seconds, -t) unlimited

              max user processes              (-u) 1024

              virtual memory          (kbytes, -v) unlimited

              file locks                      (-x) unlimited

              [12:03:01 root@S2PXDM]$







              Just wondering even though 64kb is used for 'max locked memory', we are able to lock the datastore.

              And would like to know if we are missing anything here.





              [12:03:01 root@S2PXDM $




              ------ Shared Memory Segments --------

              key        shmid      owner      perms      bytes      nattch     status

              0x5d0500ac 0          root       660        1048576    14

              0x15040012 32769      root       660        8788663088 30                locked

              0x16040012 65538      root       660        268435456  30

              0x15040013 98307      root       660        9199704880 6                 locked

              0x16040013 131076     root       660        268435456  6










              • 4. Re: Max locked memory



                Why are you running TimesTen as root? That is against security best practice and is a deprecated configuration. We strongly discourage installing and running TimesTen as root and we may actually desupport this in a future release. There really is no legitimate reason nowadays to run TimesTen as root.


                Normally, if your limit for locked memory is smaller than the total size of the datastore shared memory segment (which is given by PermSize+TempSize+LogBufMB+64, units are MB) then you will be unable to successfully load the datastore when MemoryLock=4 is specified.


                My guess (I have not tested it) is that the limit on locked memory is not enforced for 'root' and so you are able to load the datastore successfulyl even though your configuration is incorrect. Yet another reason not to use root.



                1 person found this helpful
                • 5. Re: Max locked memory

                  Hello Chris,

                  Thanks for the update.



                  Yes, we are planning to move to a non-root user as instance administrator.

                  As our application started using TimesTen from Release 5.x, we had some dependencies with our application which needed root privilege. And some monitoring tools which were developed also required root privilege. We are currently investigating the possibilities for moving to a non-root user. It is in our pipe line.





                  • 6. Re: Max locked memory

                    Hi Chris,

                    Another question we have is regarding the total space used for locking.



                    For example, Im having a datastore with PERMSIZE of 8GB, ie PERM_ALLOCATED_SIZE is 8GB

                    The actual datastore size used size is 3GB, ie PERM_IN_USE_SIZE is 3GB

                    If we have enabled memory lock (MemoryLock=4), is it that all the 8GB will be locked by TimesTen or only 3GB is locked by TimesTen and the remaining 5 GB is available for other application processes?





                    • 7. Re: Max locked memory

                      TimesTen always allocates the entire shared memory segment (PermSize + TempSize + LogBufMB + 64) up front regardless of whether it is locked or not. If we are told to lock the segment then we of course lock the entire segment. If your machine does not have enough memory to support locking the TT segment while still leaving memory for other applications then either your database is too large (i.e. larger than you need) or your machine has insufficient RAM. It is of course possible to run without MemoryLock but then you leave yourself exposed to  performance impacts from virtual memory paging. We always recommend using MemoryLock=4 (or huge pages) for a production database.



                      • 8. Re: Max locked memory

                        Hi Chris,


                        You said >>>If we are told to lock the segment then we of course lock the entire segment. >>>>


                        By this, I understand that the whole 8GB is not permitted to use by any other process. Is it correct?


                        Will other processes will be able to access 5 GB from the 8GB, since PERM_IN_USE_SIZE is only 3GB




                        • 9. Re: Max locked memory

                          Hi Pratheej,


                          That question, as posed, does not really make sense. The 8GB is allocated as a single shared memory segment. Access to that segment is limited to TimesTen processes and applications. On a virtual memory OS such as Linux/Unix, Windows etc. at any moment of time only some of the memory making up this segment may actually be mapped to physical memory. This is the same as for any other type of memory in the system such as regular process memory. In general there is always memory oversubscription; the system can allocate more (virtual) memory than there is real physical memory available. The virtual memory management system tries to keep the most recently used pages of virtual memory in physical memory but by default it allows any and all pages to be paged out (saved to disk [or discarded if not modified] and the physical memory that backs the page allocated to a different virtual page for a different usage). Locking memory prevents the OS from paging it and hence ensures that all pages making up the locked region are  always in physical memory and are accessible without any (potentially large) delay due to having to wait for the contents to be paged back in.


                          So without MemoryLock potentially *all* of the memory making up the datatsore (i.e. all 8 GB) is subject to paging in a completely unpredictable manner (i.e. it is 'available for use by other processes'). This can result in unpredictable performance and unexpected I/O load on the system depending on the overall levels of memory usage, memory access patterns and the specifics of the OS memory management algorithms. The database, like any other memory in the system, is at the mercy of the OS and the current level of 'demand' for memory and there is nothing we can do to control this. This is generally an undesirable situation hence our recommendation to use MemoryLock. Using MemoryLock 'fixes' the entire datastore (all 8 GB or whatever size it is defined as) into physical memory and disallows paging of any part of it. This provides consistent and predictable response times for anything accessing the database. This does of course make this 8 GB of physical memory 'dedicated' to the TimesTen database but that is a good thing. It will only be a problem if (a) your system is not properly configured or (b) you do not have enough physical memory to support the usage of the system. If either of these is the case then the solution is to rectify the underlying issue not to turn off memory lock!


                          I hope that clarifies.





                          1 person found this helpful