6 Replies Latest reply: Jan 16, 2013 9:49 AM by 930542 RSS

    Confirm that SHMMAX kernel parameter is set > datastore size

    930542
      Hi,

      I'm getting below exception when i'm trying to create datastore. The error is complaining about failure to create shared memory segment.
      I did go through the T-10 documentation and various post on this forum, and have made sure that the OS and the DSN settings are as per the recommended values. The values meets(or rather exceeds) the minimum values.

      The values are based on below formulas:
      kernel.shmmax (in bytes) = Should be >= PermSize + TempSize + LogBufMB +~50MB overhead
      kernel.shmall (in pages) = Should be >= ceil ( SHMMAX / PAGE_SIZE)
      SwapTotal (in KB) = Swap space should generally be at least 1.5x total physical memory size. It should be sufficient to allow allocation of a large enough shared memory segment
      TempSize (in MB) = Should be >= 40 MB + ceiling( PermSize / 8 MB)


      I have pasted below following things. Please, can any one help me resolve this issue.
      - exception in ttmessg.log
      - output of cat /etc/sysctl.conf
      - sys.odbc.ini datastore configuration
      - cat /proc/meminfo
      - Swap space
      - output of ttstatus
      - output of ttversion


      [oracle@ttnode1 info]$ ttdaemonadmin -stop
      TimesTen Daemon stopped.
      [oracle@ttnode1 info]$ ttdaemonadmin -start
      TimesTen Daemon startup OK.
      [oracle@ttnode1 info]$ ttisql "dsn=tt"

      Copyright (c) 1996-2011, Oracle. All rights reserved.
      Type ? or "help" for help, type "exit" to quit ttIsql.



      connect "dsn=tt";
      *836: Cannot create data store shared-memory segment, error 22*
      The command failed.
      Done

      The exception in ttmesg.log
      [oracle@ttnode1 info]$ tail -f ttmesg.log
      2013-01-14 17:42:49.98 Warn: : 6533: TimesTen Daemon Release 11.2.2.4.0.tt started.
      2013-01-14 17:42:59.45 Err : : 6533: TT14000: TimesTen daemon internal error: Error 22 creating shared segment, KEY 0x0200778b
      2013-01-14 17:42:59.45 Err : : 6533: -- OS reports invalid shared segment size
      2013-01-14 17:42:59.45 Err : : 6533: -- Confirm that SHMMAX kernel parameter is set > datastore size
      2013-01-14 17:45:32.67 Warn: : 6533: request id check 'MKB9+MF7V7DybTjglCEC/10d5/N7FCEFtayvZF4a9xQ=' failed (-5)
      2013-01-14 17:47:18.45 Warn: : 6533: request id check 'u6GF7bBtmRQCXKYMAWa7LhwCv1GtIG8HXdW0GLRLHEU=' failed (-5)
      2013-01-14 17:47:41.62 Warn: : 6533: request id check 'nNAzUc/plNUStN85qP9htbsz/E5psODMEeRYLmE/n50=' failed (-5)
      2013-01-14 23:15:34.58 Warn: : 8590: TimesTen Daemon Release 11.2.2.4.0.tt started.
      2013-01-14 23:15:43.44 Err : : 8590: TT14000: TimesTen daemon internal error: Error 22 creating shared segment, KEY 0x02008294
      2013-01-14 23:15:43.44 Err : : 8590: -- OS reports invalid shared segment size
      2013-01-14 23:15:43.44 Err : : 8590: -- Confirm that SHMMAX kernel parameter is set > datastore size
      2013-01-14 23:16:11.23 Warn: : 8590: request id check 'v4TJ+ZyP3KRvGxfkguLgxNyRUdtfw/MMFrvR2tWMUBk=' failed (-5)
      2013-01-14 23:18:01.34 Warn: : 8590: request id check 'p4DVyvW75Fjlo3dQ5f8OFbk4qpolqj8gFyQhqfW4ekQ=' failed (-5)
      2013-01-14 23:19:50.28 Warn: : 8590: request id check 'DPQ8gSYcljQtkuvx6ccI5a5OGcjXklItV4Mj3Cf0DmA=' failed (-5)
      2013-01-14 23:41:01.97 Warn: : 8820: TimesTen Daemon Release 11.2.2.4.0.tt started.
      *2013-01-14 23:41:10.10 Err : : 8820: TT14000: TimesTen daemon internal error: Error 22 creating shared segment, KEY 0x02008294*
      *2013-01-14 23:41:10.10 Err : : 8820: -- OS reports invalid shared segment size*
      *2013-01-14 23:41:10.10 Err : : 8820: -- Confirm that SHMMAX kernel parameter is set > datastore size*
      2013-01-14 23:42:20.31 Warn: : 8820: request id check 'sHRtjK5AUKI/RYPN6Mf6ivOPeGVFysbXT4eU9dfBZPw=' failed (-5)
      2013-01-14 23:43:24.14 Warn: : 8905: TimesTen Daemon Release 11.2.2.4.0.tt started.
      2013-01-14 23:43:29.64 Err : : 8905: TT14000: TimesTen daemon internal error: Error 22 creating shared segment, KEY 0x02008294
      2013-01-14 23:43:29.64 Err : : 8905: -- OS reports invalid shared segment size
      2013-01-14 23:43:29.64 Err : : 8905: -- Confirm that SHMMAX kernel parameter is set > datastore size

      Here is the OS setting.

      *[oracle@ttnode1 ~]$ cat /etc/sysctl.conf*
      # Kernel sysctl configuration file for Red Hat Linux
      #
      # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
      # sysctl.conf(5) for more details.

      # Controls IP packet forwarding
      net.ipv4.ip_forward = 0

      # Controls source route verification
      net.ipv4.conf.default.rp_filter = 1

      # Do not accept source routing
      net.ipv4.conf.default.accept_source_route = 0

      # Controls the System Request debugging functionality of the kernel
      kernel.sysrq = 0

      # Controls whether core dumps will append the PID to the core filename
      # Useful for debugging multi-threaded applications
      kernel.core_uses_pid = 1

      # Controls the use of TCP syncookies
      net.ipv4.tcp_syncookies = 1

      # Controls the maximum size of a message, in bytes
      kernel.msgmnb = 65536

      # Controls the default maxmimum size of a mesage queue
      kernel.msgmax = 65536

      # Controls the maximum shared segment size, in bytes
      kernel.shmmax = 68719476736

      # Controls the maximum number of shared memory segments, in pages
      kernel.shmall = 4294967296

      Here's the DSN setting

      * Datastore configuration in sys.odbc.ini file*
      *[oracle@ttnode1 info]$ grep -v ^# sys.odbc.ini | sed '/^ *$/d'
      [ODBC Data Sources]
      TT=TimesTen 11.2.2 Driver
      [TT]
      Driver=/home/oracle/TimesTen/tt/lib/libtten.so
      DataStore=/home/oracle/TimesTen/tt/info/DStore/itt_11224
      DatabaseCharacterSet=AL32UTF8
      PermSize=128
      TempSize=96
      ReplicationApplyOrdering=0
      ReplicationParallelism=4
      LogBufParallelism=4
      LogBufMB=32
      LogFileSize=32
      LogFlushMethod=2

      *[ice@ttnode1 ~]$ cat /proc/meminfo*
      MemTotal: 3865624 kB
      MemFree: 2287936 kB
      Buffers: 9744 kB
      Cached: 189340 kB
      SwapCached: 62588 kB
      Active: 739244 kB
      Inactive: 796816 kB
      HighTotal: 0 kB
      HighFree: 0 kB
      LowTotal: 3865624 kB
      LowFree: 2287936 kB
      SwapTotal: 2031608 kB
      SwapFree: 1755480 kB
      Dirty: 316 kB
      Writeback: 0 kB
      AnonPages: 1310800 kB
      Mapped: 24260 kB
      Slab: 12952 kB
      PageTables: 8712 kB
      NFS_Unstable: 0 kB
      Bounce: 0 kB
      CommitLimit: 3964420 kB
      Committed_AS: 2886040 kB
      VmallocTotal: 34359738367 kB
      VmallocUsed: 1616 kB
      VmallocChunk: 34359736599 kB
      HugePages_Total: 0
      HugePages_Free: 0
      HugePages_Rsvd: 0
      Hugepagesize: 2048 kB

      *[ice@ttnode1 ~]$ grep SwapTotal /proc/meminfo*
      SwapTotal: 4128752 kB


      Earlier the swap space was 2031608 KB, but snice it shuold be 1.5 times the physical memory, i bumped it up to its current value


      *[oracle@ttnode1 info]$ ttstatus*
      TimesTen status report as of Mon Jan 14 17:47:32 2013

      Daemon pid 6533 port 53398 instance tt
      TimesTen server pid 6550 started on port 53399
      ------------------------------------------------------------------------
      Accessible by group oinstall
      End of report

      *[oracle@ttnode1 info]$ ttversion*
      TimesTen Release 11.2.2.4.0 (64 bit Linux/x86_64) (tt:53398) 2012-09-24T08:28:05Z
      Instance admin: oracle
      Instance home directory: /homankse/oracle/TimesTen/tt
      Group owner: oinstall
      Daemon home directory: /home/oracle/TimesTen/tt/info

      Thanks,
      Ved
        • 1. Re: Confirm that SHMMAX kernel parameter is set > datastore size
          Rajeshp-Oracle
          What is the output of below

          ipcs -a
          ttstatus

          Regards
          Rajesh
          • 2. Re: Confirm that SHMMAX kernel parameter is set > datastore size
            Jspalmer-Oracle
            The DSN setings you are using are tiny:

            PermSize=128
            TempSize=96
            LogBufMB=32

            These figures are in MB not GB, so you are affectively creating a shared memory segment of around 300MB. Which makes it even less explicable why it cannot create a shared memory segment with these attributes when your kernel configurations seem quite adequate.

            Please check that TimesTen is reading from the expected sys.odbc.ini file (and so is not trying to use a different DSN configuration with, perhaps, much higher values). It may be helpful too to add the -verbose option to your <TT_install_dir>/info/ttendaemon.options file, so that the ttmesg.log contains INFO level messages as well. This will provide more detail on the values being used. Lastly, please also check the ulimit settings which would be being used by the TimesTen processes - you can use

            ttStatus -debug | grep RLIMIT

            to see these.
            • 3. Re: Confirm that SHMMAX kernel parameter is set > datastore size
              930542
              Hi, Thanks for the reply.

              Here's more info that was asked for.

              ttStatus
              TimesTen status report as of Tue Jan 15 16:40:28 2013

              Daemon pid 8905 port 53398 instance tt
              TimesTen server pid 8919 started on port 53399
              ------------------------------------------------------------------------
              Accessible by group oinstall
              End of report

              ttStatus -debug | grep RLIMIT

              RLIMIT_CORE: cur 0, max unlimited
              RLIMIT_CPU: cur unlimited, max unlimited
              RLIMIT_DATA: cur unlimited, max unlimited
              RLIMIT_FSIZE: cur unlimited, max unlimited
              RLIMIT_NOFILE: cur 1024, max 1024
              RLIMIT_STACK: cur 10485760, max unlimited
              RLIMIT_AS: cur unlimited, max unlimited
              RLIMIT_LOCKS: cur unlimited, max unlimited
              RLIMIT_MEMLOCK: cur 32768, max 32768
              RLIMIT_RSS: cur unlimited, max unlimited
              RLIMIT_MSGQUEUE: cur 819200, max 819200
              RLIMIT_NPROC: cur 30720, max 30720

              ipcs -a

              ------ Shared Memory Segments --------
              key shmid owner perms bytes nattch status
              0x00000000 3211265 root 644 80 2
              0x00000000 3244035 root 644 16384 2
              0x00000000 3276804 root 644 280 2
              0x47007bf9 3801093 oracle 660 1048576 1

              ------ Semaphore Arrays --------
              key semid owner perms nsems
              0x47007bf9 524289 oracle 660 1

              ------ Message Queues --------
              key msqid owner perms used-bytes messages



              Again, thanks for helping.
              -Ved
              • 4. Re: Confirm that SHMMAX kernel parameter is set > datastore size
                930542
                Hi,

                On continuing my debugging steps, if checked the limits for IPC facility.

                ipcs -m -l

                ------ Shared Memory Limits --------
                max number of segments = 4096
                max seg size (kbytes) = 32768
                max total shared memory (kbytes) = 8388608
                min seg size (bytes) = 1

                This says that "max number of segments = 4096", where as the kernel parameter is defined as "kernel.shmall = 4294967296".

                Am i correct in understanding that "max number of segments" is same as "kernel.shmall" and the values should be same.

                Also, it says that "max seg size (kbytes) = 32768" ie around 32MB, where as the kernel parameter is defined as "kernel.shmmax = 68719476736" i.e around 64BG.
                and since, the DSN is requesting to create shared memory segment of 128MB, which is greater that 32MB(as per max seg size (kbytes) = 32768), Timesten is throwing the error.


                Thanks everyone for helping me out.
                _ved                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                • 5. Re: Confirm that SHMMAX kernel parameter is set > datastore size
                  Rajeshp-Oracle
                  I think this is the relation

                  max number of segments = 4096 // SHMMNI     
                  max seg size (kbytes) = 32768 // SHMMAX
                  max total shared memory (kbytes) = 8388608 // SHMALL
                  min seg size (bytes) = 1

                  Looking into your output of ipcs -l , may be the changes you have made in sysctl.conf file is not reflecting in the ipcs -l output(still showing the default values I guess) . Did you do reboot the machine or done sysctl -p ?


                  Regards
                  Rajesh
                  • 6. Re: Confirm that SHMMAX kernel parameter is set > datastore size
                    930542
                    Hello Everyone,

                    Thanks for the help provided. The issue is now resolved.

                    -Ved