0 Replies Latest reply on Oct 7, 2018 1:44 PM by 2986706

    High CPU utilization on RHEL 7.2 OS - sbWaitLatchGet consumes 60% of application CPU      Printable View           Update

    2986706

      We have observed performance degrade on RHEL 7.2 environment. As per FlameGraph report a TT library function i.e. sbWaitLatchGet is consuming more than 60% CPU. It is resulting application performance.

       

      Below are same in RHEL 6.2 and RHEL 7.2

      1. Application

      2. Timesten software and TT config

      3. Oracle DB and its software is same

      4. Network.

      5. TPS - call flows and number of requests

      6. All config is same expect OS

       

      We have used perf tool and FlameGraph and observed that sbWaitLatchGet() is called more times in RHEL 7.2. Please find below call graphs and CPU usage for RHEL 6.2 and 7.2.  It looks like most of the CPI is spent for latching/locking. Is there a way to reduce it?

       

      = Profiling status

      Thread 11140 (CdbTTDbMgr)

      Rate 10 samples/second

      Total 4267 samples taken in 1062 seconds

       

      = Profiling results

      1. Number of profiling samples in this function

      2. Percentage of profiling samples in this function

      3. Number of profiling samples in its callees

      4. Number of profiling samples in this function and its callees

      5. Percentage of profiling samples in this function and its callees

      6. Function name

       

          3050   71.48%        5     3055   71.60%  sbWaitLatchGet

            96    2.25%     1910     2006   47.01%  sb_sqlCurOpenOFC

            94    2.20%        8      102    2.39%  SQLExecute

            64    1.50%        0       64    1.50%  write

            44    1.03%        0       44    1.03%  read

            28    0.66%       79      107    2.51%  sb_sqlPassThroughInParamTypesSet

            26    0.61%       25       51    1.20%  sb_xactCommitQ

       

      # Children      Self       Samples  Command   Shared Object           Symbol                                                                                        

      # ........  ........  ............  ........  ...................... 

      #

          62.37%    61.38%         10628  cdbShMgr  libtten.so              [.] sbWaitLatchGet                                                                                                                                                                                                                                                                                                        

                  |

                  ---sbWaitLatchGet

                     |         

                     |--37.64%-- sbEePassThroughOpen

                     |          sbEeStreamOpen

                     |          sbSqlCurOpenOFC

                     |          sb_sqlCurOpenOFC

                     |          ttpsdofc

                     |          ttpsdnal

                     |         

                     |--25.31%-- sbEeStreamOpen

                     |          sbSqlCurOpenOFC

                     |          sb_sqlCurOpenOFC

                     |          ttpsdofc

                     |          ttpsdnal

                     |          |         

                     |           --100.00%-- pevm_EXIM

                     |         

                     |--20.81%-- sbEeStreamClose

                     |          sbSqlCurClose

                     |          sbCursorClose

                     |          sb_curCloseQ

                     |          tpscFastCloseCursor

                     |          kgttscFastCloseCursor

                     |          kgscReleaseACursor

                     |          pevm_EXIM

                     |          pfrinstr_EXIM

                     |          pfrrun_no_tool

                     |          pfrrun

                     |          plsql_run

                     |          peicnt

                     |          ttxexe

                     |          ttsmain

                     |         

                     |--13.97%-- sbSqlCmdExec

                     |          sb_sqlExecQ

                     |          ttpsdiexn

                     |          ttpsdnal

                     |          pevm_EXIM

                     |          pfrinstr_EXIM

                     |          pfrrun_no_tool

                     |          pfrrun

                     |          plsql_run

                     |         

                     |--2.20%-- ttBDbXactSavePoint

                     |          sbEePassThroughOpen

                     |          sbEeStreamOpen

                     |          sbSqlCurOpenOFC

                     |          sb_sqlCurOpenOFC

                     |          ttpsdofc

                     |          ttpsdnal

                      --0.06%-- [...]

       

       

       

      TT Config:

       

      Command> call ttConfiguration();

      < CacheAwtMethod, 1 >

      < CacheAwtParallelism, 8 >

      < CacheGridEnable, 1 >

      < CacheGridMsgWait, 60 >

      < CkptFrequency, 600 >

      < CkptLogVolume, 0 >

      < CkptRate, 20 >

      < CkptReadThreads, 1 >

      < CommitBufferSizeMax, 2048 >

      < ConnectionCharacterSet, US7ASCII >

      < ConnectionName, cache1a >

      < Connections, 200 >

      < DDLCommitBehavior, 0 >

      < DDLReplicationAction, INCLUDE >

      < DDLReplicationLevel, 2 >

      < DataBaseCharacterSet, WE8MSWIN1252 >

      < DataStore, /data/timesten/datastore/instancea/cache1A_dst >

      < DynamicLoadEnable, 1 >

      < DuplicateBindMode, 0 >

      < DurableCommits, 0 >

      < DynamicLoadErrorMode, 0 >

      < Isolation, 1 >

      < RangeIndexType, 1 >

      < LockLevel, 0 >

      < LockWait, 5 >

      < LogAutoTruncate, 1 >

      < LogBufMB, 256 >

      < LogBufParallelism, 8 >

      < LogDir, /data/timesten/ttlogs/instancea >

      < LogFileSize, 256 >

      < LogFlushMethod, 1 >

      < LogPurge, 1 >

      < Logging, 1 >

      < MemoryLock, 4 >

      < NLS_LENGTH_SEMANTICS, BYTE >

      < NLS_NCHAR_CONV_EXCP, 0 >

      < NLS_SORT, BINARY >

      < OracleNetServiceName, oracledb >

      < PLSCOPE_SETTINGS, IDENTIFIERS:NONE >

      < PLSQL, 1 >

      < PLSQL_CCFLAGS, <NULL> >

      < PLSQL_CODE_TYPE, INTERPRETED >

      < PLSQL_CONN_MEM_LIMIT, 100 >

      < PLSQL_MEMORY_ADDRESS, 0x0000007fa0000000 >

      < PLSQL_MEMORY_SIZE, 150 >

      < PLSQL_OPTIMIZE_LEVEL, 2 >

      < PLSQL_TIMEOUT, 24 >

      < PassThrough, 0 >

      < PermSize, 40960 >

      < PermWarnThreshold, 90 >

      < Preallocate, 0 >

      < PrivateCommands, 1 >

      < QueryThreshold, 0 >

      < RACCallback, 1 >

      < ReceiverThreads, 1 >

      < RecoveryThreads, 2 >

      < ReplicationApplyOrdering, 0 >

      < ReplicationParallelism, 1 >

      < ReplicationTrack, 255 >

      < SQLQueryTimeout, 24 >

      < TempSize, 4096 >

      < TempWarnThreshold, 90 >

      < Temporary, 0 >

      < TypeMode, 0 >

      < UID, CDBTT_SYS >

      65 rows found.