This discussion is archived
6 Replies Latest reply: Jan 16, 2013 7:49 AM by 930542 RSS

Confirm that SHMMAX kernel parameter is set > datastore size

930542 Newbie
Currently Being Moderated
Hi,

I'm getting below exception when i'm trying to create datastore. The error is complaining about failure to create shared memory segment.
I did go through the T-10 documentation and various post on this forum, and have made sure that the OS and the DSN settings are as per the recommended values. The values meets(or rather exceeds) the minimum values.

The values are based on below formulas:
kernel.shmmax (in bytes) = Should be >= PermSize + TempSize + LogBufMB +~50MB overhead
kernel.shmall (in pages) = Should be >= ceil ( SHMMAX / PAGE_SIZE)
SwapTotal (in KB) = Swap space should generally be at least 1.5x total physical memory size. It should be sufficient to allow allocation of a large enough shared memory segment
TempSize (in MB) = Should be >= 40 MB + ceiling( PermSize / 8 MB)


I have pasted below following things. Please, can any one help me resolve this issue.
- exception in ttmessg.log
- output of cat /etc/sysctl.conf
- sys.odbc.ini datastore configuration
- cat /proc/meminfo
- Swap space
- output of ttstatus
- output of ttversion


[oracle@ttnode1 info]$ ttdaemonadmin -stop
TimesTen Daemon stopped.
[oracle@ttnode1 info]$ ttdaemonadmin -start
TimesTen Daemon startup OK.
[oracle@ttnode1 info]$ ttisql "dsn=tt"

Copyright (c) 1996-2011, Oracle. All rights reserved.
Type ? or "help" for help, type "exit" to quit ttIsql.



connect "dsn=tt";
*836: Cannot create data store shared-memory segment, error 22*
The command failed.
Done

The exception in ttmesg.log
[oracle@ttnode1 info]$ tail -f ttmesg.log
2013-01-14 17:42:49.98 Warn: : 6533: TimesTen Daemon Release 11.2.2.4.0.tt started.
2013-01-14 17:42:59.45 Err : : 6533: TT14000: TimesTen daemon internal error: Error 22 creating shared segment, KEY 0x0200778b
2013-01-14 17:42:59.45 Err : : 6533: -- OS reports invalid shared segment size
2013-01-14 17:42:59.45 Err : : 6533: -- Confirm that SHMMAX kernel parameter is set > datastore size
2013-01-14 17:45:32.67 Warn: : 6533: request id check 'MKB9+MF7V7DybTjglCEC/10d5/N7FCEFtayvZF4a9xQ=' failed (-5)
2013-01-14 17:47:18.45 Warn: : 6533: request id check 'u6GF7bBtmRQCXKYMAWa7LhwCv1GtIG8HXdW0GLRLHEU=' failed (-5)
2013-01-14 17:47:41.62 Warn: : 6533: request id check 'nNAzUc/plNUStN85qP9htbsz/E5psODMEeRYLmE/n50=' failed (-5)
2013-01-14 23:15:34.58 Warn: : 8590: TimesTen Daemon Release 11.2.2.4.0.tt started.
2013-01-14 23:15:43.44 Err : : 8590: TT14000: TimesTen daemon internal error: Error 22 creating shared segment, KEY 0x02008294
2013-01-14 23:15:43.44 Err : : 8590: -- OS reports invalid shared segment size
2013-01-14 23:15:43.44 Err : : 8590: -- Confirm that SHMMAX kernel parameter is set > datastore size
2013-01-14 23:16:11.23 Warn: : 8590: request id check 'v4TJ+ZyP3KRvGxfkguLgxNyRUdtfw/MMFrvR2tWMUBk=' failed (-5)
2013-01-14 23:18:01.34 Warn: : 8590: request id check 'p4DVyvW75Fjlo3dQ5f8OFbk4qpolqj8gFyQhqfW4ekQ=' failed (-5)
2013-01-14 23:19:50.28 Warn: : 8590: request id check 'DPQ8gSYcljQtkuvx6ccI5a5OGcjXklItV4Mj3Cf0DmA=' failed (-5)
2013-01-14 23:41:01.97 Warn: : 8820: TimesTen Daemon Release 11.2.2.4.0.tt started.
*2013-01-14 23:41:10.10 Err : : 8820: TT14000: TimesTen daemon internal error: Error 22 creating shared segment, KEY 0x02008294*
*2013-01-14 23:41:10.10 Err : : 8820: -- OS reports invalid shared segment size*
*2013-01-14 23:41:10.10 Err : : 8820: -- Confirm that SHMMAX kernel parameter is set > datastore size*
2013-01-14 23:42:20.31 Warn: : 8820: request id check 'sHRtjK5AUKI/RYPN6Mf6ivOPeGVFysbXT4eU9dfBZPw=' failed (-5)
2013-01-14 23:43:24.14 Warn: : 8905: TimesTen Daemon Release 11.2.2.4.0.tt started.
2013-01-14 23:43:29.64 Err : : 8905: TT14000: TimesTen daemon internal error: Error 22 creating shared segment, KEY 0x02008294
2013-01-14 23:43:29.64 Err : : 8905: -- OS reports invalid shared segment size
2013-01-14 23:43:29.64 Err : : 8905: -- Confirm that SHMMAX kernel parameter is set > datastore size

Here is the OS setting.

*[oracle@ttnode1 ~]$ cat /etc/sysctl.conf*
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

Here's the DSN setting

* Datastore configuration in sys.odbc.ini file*
*[oracle@ttnode1 info]$ grep -v ^# sys.odbc.ini | sed '/^ *$/d'
[ODBC Data Sources]
TT=TimesTen 11.2.2 Driver
[TT]
Driver=/home/oracle/TimesTen/tt/lib/libtten.so
DataStore=/home/oracle/TimesTen/tt/info/DStore/itt_11224
DatabaseCharacterSet=AL32UTF8
PermSize=128
TempSize=96
ReplicationApplyOrdering=0
ReplicationParallelism=4
LogBufParallelism=4
LogBufMB=32
LogFileSize=32
LogFlushMethod=2

*[ice@ttnode1 ~]$ cat /proc/meminfo*
MemTotal: 3865624 kB
MemFree: 2287936 kB
Buffers: 9744 kB
Cached: 189340 kB
SwapCached: 62588 kB
Active: 739244 kB
Inactive: 796816 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 3865624 kB
LowFree: 2287936 kB
SwapTotal: 2031608 kB
SwapFree: 1755480 kB
Dirty: 316 kB
Writeback: 0 kB
AnonPages: 1310800 kB
Mapped: 24260 kB
Slab: 12952 kB
PageTables: 8712 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 3964420 kB
Committed_AS: 2886040 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1616 kB
VmallocChunk: 34359736599 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

*[ice@ttnode1 ~]$ grep SwapTotal /proc/meminfo*
SwapTotal: 4128752 kB


Earlier the swap space was 2031608 KB, but snice it shuold be 1.5 times the physical memory, i bumped it up to its current value


*[oracle@ttnode1 info]$ ttstatus*
TimesTen status report as of Mon Jan 14 17:47:32 2013

Daemon pid 6533 port 53398 instance tt
TimesTen server pid 6550 started on port 53399
------------------------------------------------------------------------
Accessible by group oinstall
End of report

*[oracle@ttnode1 info]$ ttversion*
TimesTen Release 11.2.2.4.0 (64 bit Linux/x86_64) (tt:53398) 2012-09-24T08:28:05Z
Instance admin: oracle
Instance home directory: /homankse/oracle/TimesTen/tt
Group owner: oinstall
Daemon home directory: /home/oracle/TimesTen/tt/info

Thanks,
Ved
  • 1. Re: Confirm that SHMMAX kernel parameter is set > datastore size
    rajeshp Explorer
    Currently Being Moderated
    What is the output of below

    ipcs -a
    ttstatus

    Regards
    Rajesh
  • 2. Re: Confirm that SHMMAX kernel parameter is set > datastore size
    jspalmer Journeyer
    Currently Being Moderated
    The DSN setings you are using are tiny:

    PermSize=128
    TempSize=96
    LogBufMB=32

    These figures are in MB not GB, so you are affectively creating a shared memory segment of around 300MB. Which makes it even less explicable why it cannot create a shared memory segment with these attributes when your kernel configurations seem quite adequate.

    Please check that TimesTen is reading from the expected sys.odbc.ini file (and so is not trying to use a different DSN configuration with, perhaps, much higher values). It may be helpful too to add the -verbose option to your <TT_install_dir>/info/ttendaemon.options file, so that the ttmesg.log contains INFO level messages as well. This will provide more detail on the values being used. Lastly, please also check the ulimit settings which would be being used by the TimesTen processes - you can use

    ttStatus -debug | grep RLIMIT

    to see these.
  • 3. Re: Confirm that SHMMAX kernel parameter is set > datastore size
    930542 Newbie
    Currently Being Moderated
    Hi, Thanks for the reply.

    Here's more info that was asked for.

    ttStatus
    TimesTen status report as of Tue Jan 15 16:40:28 2013

    Daemon pid 8905 port 53398 instance tt
    TimesTen server pid 8919 started on port 53399
    ------------------------------------------------------------------------
    Accessible by group oinstall
    End of report

    ttStatus -debug | grep RLIMIT

    RLIMIT_CORE: cur 0, max unlimited
    RLIMIT_CPU: cur unlimited, max unlimited
    RLIMIT_DATA: cur unlimited, max unlimited
    RLIMIT_FSIZE: cur unlimited, max unlimited
    RLIMIT_NOFILE: cur 1024, max 1024
    RLIMIT_STACK: cur 10485760, max unlimited
    RLIMIT_AS: cur unlimited, max unlimited
    RLIMIT_LOCKS: cur unlimited, max unlimited
    RLIMIT_MEMLOCK: cur 32768, max 32768
    RLIMIT_RSS: cur unlimited, max unlimited
    RLIMIT_MSGQUEUE: cur 819200, max 819200
    RLIMIT_NPROC: cur 30720, max 30720

    ipcs -a

    ------ Shared Memory Segments --------
    key shmid owner perms bytes nattch status
    0x00000000 3211265 root 644 80 2
    0x00000000 3244035 root 644 16384 2
    0x00000000 3276804 root 644 280 2
    0x47007bf9 3801093 oracle 660 1048576 1

    ------ Semaphore Arrays --------
    key semid owner perms nsems
    0x47007bf9 524289 oracle 660 1

    ------ Message Queues --------
    key msqid owner perms used-bytes messages



    Again, thanks for helping.
    -Ved
  • 4. Re: Confirm that SHMMAX kernel parameter is set > datastore size
    930542 Newbie
    Currently Being Moderated
    Hi,

    On continuing my debugging steps, if checked the limits for IPC facility.

    ipcs -m -l

    ------ Shared Memory Limits --------
    max number of segments = 4096
    max seg size (kbytes) = 32768
    max total shared memory (kbytes) = 8388608
    min seg size (bytes) = 1

    This says that "max number of segments = 4096", where as the kernel parameter is defined as "kernel.shmall = 4294967296".

    Am i correct in understanding that "max number of segments" is same as "kernel.shmall" and the values should be same.

    Also, it says that "max seg size (kbytes) = 32768" ie around 32MB, where as the kernel parameter is defined as "kernel.shmmax = 68719476736" i.e around 64BG.
    and since, the DSN is requesting to create shared memory segment of 128MB, which is greater that 32MB(as per max seg size (kbytes) = 32768), Timesten is throwing the error.


    Thanks everyone for helping me out.
    _ved                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 5. Re: Confirm that SHMMAX kernel parameter is set > datastore size
    rajeshp Explorer
    Currently Being Moderated
    I think this is the relation

    max number of segments = 4096 // SHMMNI     
    max seg size (kbytes) = 32768 // SHMMAX
    max total shared memory (kbytes) = 8388608 // SHMALL
    min seg size (bytes) = 1

    Looking into your output of ipcs -l , may be the changes you have made in sysctl.conf file is not reflecting in the ipcs -l output(still showing the default values I guess) . Did you do reboot the machine or done sysctl -p ?


    Regards
    Rajesh
  • 6. Re: Confirm that SHMMAX kernel parameter is set > datastore size
    930542 Newbie
    Currently Being Moderated
    Hello Everyone,

    Thanks for the help provided. The issue is now resolved.

    -Ved

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points