High CPU utilization on RHEL 7.2 OS - sbWaitLatchGet consumes 60% of application CPU Printable View Update
We have observed performance degrade on RHEL 7.2 environment. As per FlameGraph report a TT library function i.e. sbWaitLatchGet is consuming more than 60% CPU. It is resulting application performance.
Below are same in RHEL 6.2 and RHEL 7.2
1. Application
2. Timesten software and TT config
3. Oracle DB and its software is same
4. Network.
5. TPS - call flows and number of requests
6. All config is same expect OS
We have used perf tool and FlameGraph and observed that sbWaitLatchGet() is called more times in RHEL 7.2. Please find below call graphs and CPU usage for RHEL 6.2 and 7.2. It looks like most of the CPI is spent for latching/locking. Is there a way to reduce it?
= Profiling status
Thread 11140 (CdbTTDbMgr)
Rate 10 samples/second
Total 4267 samples taken in 1062 seconds
= Profiling results
1. Number of profiling samples in this function
2. Percentage of profiling samples in this function
3. Number of profiling samples in its callees
4. Number of profiling samples in this function and its callees
5. Percentage of profiling samples in this function and its callees
6. Function name
3050 71.48% 5 3055 71.60% sbWaitLatchGet
96 2.25% 1910 2006 47.01% sb_sqlCurOpenOFC
94 2.20% 8 102 2.39% SQLExecute
64 1.50% 0 64 1.50% write
44 1.03% 0 44 1.03% read
28 0.66% 79 107 2.51% sb_sqlPassThroughInParamTypesSet
26 0.61% 25 51 1.20% sb_xactCommitQ
# Children Self Samples Command Shared Object Symbol
# ........ ........ ............ ........ ......................
#
62.37% 61.38% 10628 cdbShMgr libtten.so [.] sbWaitLatchGet
|
---sbWaitLatchGet
|
|--37.64%-- sbEePassThroughOpen
| sbEeStreamOpen
| sbSqlCurOpenOFC
| sb_sqlCurOpenOFC
| ttpsdofc
| ttpsdnal
|
|--25.31%-- sbEeStreamOpen
| sbSqlCurOpenOFC
| sb_sqlCurOpenOFC
| ttpsdofc
| ttpsdnal
| |
| --100.00%-- pevm_EXIM
|
|--20.81%-- sbEeStreamClose
| sbSqlCurClose
| sbCursorClose
| sb_curCloseQ
| tpscFastCloseCursor
| kgttscFastCloseCursor
| kgscReleaseACursor
| pevm_EXIM
| pfrinstr_EXIM
| pfrrun_no_tool
| pfrrun
| plsql_run
| peicnt
| ttxexe
| ttsmain
|
|--13.97%-- sbSqlCmdExec
| sb_sqlExecQ
| ttpsdiexn
| ttpsdnal
| pevm_EXIM
| pfrinstr_EXIM
| pfrrun_no_tool
| pfrrun
| plsql_run
|
|--2.20%-- ttBDbXactSavePoint
| sbEePassThroughOpen
| sbEeStreamOpen
| sbSqlCurOpenOFC
| sb_sqlCurOpenOFC
| ttpsdofc
| ttpsdnal
--0.06%-- [...]
TT Config:
Command> call ttConfiguration();
< CacheAwtMethod, 1 >
< CacheAwtParallelism, 8 >
< CacheGridEnable, 1 >
< CacheGridMsgWait, 60 >
< CkptFrequency, 600 >
< CkptLogVolume, 0 >
< CkptRate, 20 >
< CkptReadThreads, 1 >
< CommitBufferSizeMax, 2048 >
< ConnectionCharacterSet, US7ASCII >
< ConnectionName, cache1a >
< Connections, 200 >
< DDLCommitBehavior, 0 >
< DDLReplicationAction, INCLUDE >
< DDLReplicationLevel, 2 >
< DataBaseCharacterSet, WE8MSWIN1252 >
< DataStore, /data/timesten/datastore/instancea/cache1A_dst >
< DynamicLoadEnable, 1 >
< DuplicateBindMode, 0 >
< DurableCommits, 0 >
< DynamicLoadErrorMode, 0 >
< Isolation, 1 >
< RangeIndexType, 1 >
< LockLevel, 0 >
< LockWait, 5 >
< LogAutoTruncate, 1 >
< LogBufMB, 256 >
< LogBufParallelism, 8 >
< LogDir, /data/timesten/ttlogs/instancea >
< LogFileSize, 256 >
< LogFlushMethod, 1 >
< LogPurge, 1 >
< Logging, 1 >
< MemoryLock, 4 >
< NLS_LENGTH_SEMANTICS, BYTE >
< NLS_NCHAR_CONV_EXCP, 0 >
< NLS_SORT, BINARY >
< OracleNetServiceName, oracledb >
< PLSCOPE_SETTINGS, IDENTIFIERS:NONE >
< PLSQL, 1 >
< PLSQL_CCFLAGS, <NULL> >
< PLSQL_CODE_TYPE, INTERPRETED >
< PLSQL_CONN_MEM_LIMIT, 100 >
< PLSQL_MEMORY_ADDRESS, 0x0000007fa0000000 >
< PLSQL_MEMORY_SIZE, 150 >
< PLSQL_OPTIMIZE_LEVEL, 2 >
< PLSQL_TIMEOUT, 24 >
< PassThrough, 0 >
< PermSize, 40960 >
< PermWarnThreshold, 90 >
< Preallocate, 0 >
< PrivateCommands, 1 >
< QueryThreshold, 0 >
< RACCallback, 1 >
< ReceiverThreads, 1 >
< RecoveryThreads, 2 >
< ReplicationApplyOrdering, 0 >
< ReplicationParallelism, 1 >
< ReplicationTrack, 255 >
< SQLQueryTimeout, 24 >
< TempSize, 4096 >
< TempWarnThreshold, 90 >
< Temporary, 0 >
< TypeMode, 0 >
< UID, CDBTT_SYS >
65 rows found.