We have observed performance degrade on RHEL 7.2 environment. As per FlameGraph report a TT library function i.e. sbWaitLatchGet is consuming more than 60% CPU. It is resulting application performance.

Below are same in RHEL 6.2 and RHEL 7.2

1. Application

2. Timesten software and TT config

3. Oracle DB and its software is same

4. Network.

5. TPS - call flows and number of requests

6. All config is same expect OS

We have used perf tool and FlameGraph and observed that sbWaitLatchGet() is called more times in RHEL 7.2. Please find below call graphs and CPU usage for RHEL 6.2 and 7.2.  It looks like most of the CPI is spent for latching/locking. Is there a way to reduce it?

= Profiling status

Thread 11140 (CdbTTDbMgr)

Rate 10 samples/second

Total 4267 samples taken in 1062 seconds

= Profiling results

1. Number of profiling samples in this function

2. Percentage of profiling samples in this function

3. Number of profiling samples in its callees

4. Number of profiling samples in this function and its callees

5. Percentage of profiling samples in this function and its callees

6. Function name

    3050   71.48%        5     3055   71.60%  sbWaitLatchGet

      96    2.25%     1910     2006   47.01%  sb_sqlCurOpenOFC

      94    2.20%        8      102    2.39%  SQLExecute

      64    1.50%        0       64    1.50%  write

      44    1.03%        0       44    1.03%  read

      28    0.66%       79      107    2.51%  sb_sqlPassThroughInParamTypesSet

      26    0.61%       25       51    1.20%  sb_xactCommitQ

# Children      Self       Samples  Command   Shared Object           Symbol                                                                                        

# ........  ........  ............  ........  ...................... 

#

    62.37%    61.38%         10628  cdbShMgr  libtten.so              [.] sbWaitLatchGet                                                                                                                                                                                                                                                                                                        

            |

            ---sbWaitLatchGet

               |         

               |--37.64%-- sbEePassThroughOpen

               |          sbEeStreamOpen

               |          sbSqlCurOpenOFC

               |          sb_sqlCurOpenOFC

               |          ttpsdofc

               |          ttpsdnal

               |         

               |--25.31%-- sbEeStreamOpen

               |          sbSqlCurOpenOFC

               |          sb_sqlCurOpenOFC

               |          ttpsdofc

               |          ttpsdnal

               |          |         

               |           --100.00%-- pevm_EXIM

               |         

               |--20.81%-- sbEeStreamClose

               |          sbSqlCurClose

               |          sbCursorClose

               |          sb_curCloseQ

               |          tpscFastCloseCursor

               |          kgttscFastCloseCursor

               |          kgscReleaseACursor

               |          pevm_EXIM

               |          pfrinstr_EXIM

               |          pfrrun_no_tool

               |          pfrrun

               |          plsql_run

               |          peicnt

               |          ttxexe

               |          ttsmain

               |         

               |--13.97%-- sbSqlCmdExec

               |          sb_sqlExecQ

               |          ttpsdiexn

               |          ttpsdnal

               |          pevm_EXIM

               |          pfrinstr_EXIM

               |          pfrrun_no_tool

               |          pfrrun

               |          plsql_run

               |         

               |--2.20%-- ttBDbXactSavePoint

               |          sbEePassThroughOpen

               |          sbEeStreamOpen

               |          sbSqlCurOpenOFC

               |          sb_sqlCurOpenOFC

               |          ttpsdofc

               |          ttpsdnal

                --0.06%-- [...]

TT Config:

Command> call ttConfiguration();

< CacheAwtMethod, 1 >

< CacheAwtParallelism, 8 >

< CacheGridEnable, 1 >

< CacheGridMsgWait, 60 >

< CkptFrequency, 600 >

< CkptLogVolume, 0 >

< CkptRate, 20 >

< CkptReadThreads, 1 >

< CommitBufferSizeMax, 2048 >

< ConnectionCharacterSet, US7ASCII >

< ConnectionName, cache1a >

< Connections, 200 >

< DDLCommitBehavior, 0 >

< DDLReplicationAction, INCLUDE >

< DDLReplicationLevel, 2 >

< DataBaseCharacterSet, WE8MSWIN1252 >

< DataStore, /data/timesten/datastore/instancea/cache1A_dst >

< DynamicLoadEnable, 1 >

< DuplicateBindMode, 0 >

< DurableCommits, 0 >

< DynamicLoadErrorMode, 0 >

< Isolation, 1 >

< RangeIndexType, 1 >

< LockLevel, 0 >

< LockWait, 5 >

< LogAutoTruncate, 1 >

< LogBufMB, 256 >

< LogBufParallelism, 8 >

< LogDir, /data/timesten/ttlogs/instancea >

< LogFileSize, 256 >

< LogFlushMethod, 1 >

< LogPurge, 1 >

< Logging, 1 >

< MemoryLock, 4 >

< NLS_LENGTH_SEMANTICS, BYTE >

< NLS_NCHAR_CONV_EXCP, 0 >

< NLS_SORT, BINARY >

< OracleNetServiceName, oracledb >

< PLSCOPE_SETTINGS, IDENTIFIERS:NONE >

< PLSQL, 1 >

< PLSQL_CCFLAGS, <NULL> >

< PLSQL_CODE_TYPE, INTERPRETED >

< PLSQL_CONN_MEM_LIMIT, 100 >

< PLSQL_MEMORY_ADDRESS, 0x0000007fa0000000 >

< PLSQL_MEMORY_SIZE, 150 >

< PLSQL_OPTIMIZE_LEVEL, 2 >

< PLSQL_TIMEOUT, 24 >

< PassThrough, 0 >

< PermSize, 40960 >

< PermWarnThreshold, 90 >

< Preallocate, 0 >

< PrivateCommands, 1 >

< QueryThreshold, 0 >

< RACCallback, 1 >

< ReceiverThreads, 1 >

< RecoveryThreads, 2 >

< ReplicationApplyOrdering, 0 >

< ReplicationParallelism, 1 >

< ReplicationTrack, 255 >

< SQLQueryTimeout, 24 >

< TempSize, 4096 >

< TempWarnThreshold, 90 >

< Temporary, 0 >

< TypeMode, 0 >

< UID, CDBTT_SYS >

65 rows found.