This discussion is archived
1 2 3 4 Previous Next 46 Replies Latest reply: Apr 19, 2012 4:27 AM by ChrisJenkins Go to original post RSS
  • 30. Re: Perfomance Tunning required in Timesten
    user10366531 Newbie
    Currently Being Moderated
    Dear Chris,

    I am unable to set MemoryLock=4, getting below error.

    11:57:33.55 Err : : 20166: TT14000: TimesTen daemon internal error: Error 12 while locking shared segment, KEY 0x120199dd ID 13434884
    11:57:33.55 Err : : 20166: -- OS reports not enough memory available to lock

    OS: uname -a
    Linux rac2 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
  • 31. Re: Perfomance Tunning required in Timesten
    user10366531 Newbie
    Currently Being Moderated
    Dear Chris,

    I just checked on net and I found that I need to check ulimit for timesten user.

    ulimit -l
    32

    So I modified the /etc/security/limit.conf
    timesten soft memlock 5242880
    timesten hard memlock 5242880

    re-login to my timesten user and checked ulimit again.

    ulimit -l
    5242880

    But still, I am getting the same error.

    [timesten@rac2 ~]$ ps -eo pmem,pcpu,args | sort -k 1 -r | less
    %MEM %CPU COMMAND
    0.0 0.2 top
    0.0 0.2 [kswapd0]
    0.0 0.0 xinetd -stayalive -pidfile /var/run/xinetd.pid
  • 32. Re: Perfomance Tunning required in Timesten
    ChrisJenkins Guru
    Currently Being Moderated
    You need to ensure that the ulimit is set consistently (maybe put it is the login profile script). Also, you ned to stop and start the TimesTen main daemon (ttDaemonAdmin -stop then ttDaemonAdmin -start) to pick up the new values from limits.conf. Should work then.

    Chris
  • 33. Re: Perfomance Tunning required in Timesten
    user10366531 Newbie
    Currently Being Moderated
    Thank you so much chris, I worked after restarting the Daemon.
    I had set ulimit to unlimited, so i am not sure how much is locked by timesten.

    Is there any way to know how much memory timesten has locked?

    Is it PermSize + TempSize + LogFileSize + LogBufMB?
  • 34. Re: Perfomance Tunning required in Timesten
    user10366531 Newbie
    Currently Being Moderated
    Dear Chris,

    About your previous message, regarding AWT write up to oracle.

    A secondary, but major, concern is this:

    Just want to understand, does writing of replication agent towards oracle impact the overall response time?
    I must be wrong but my understanding was that timesten will take writing towards oracle asynchronously for AWT cache groups.
    It will not impact my application performance.

    Sorry, but just want to understand the behavior.
  • 35. Re: Perfomance Tunning required in Timesten
    ChrisJenkins Guru
    Currently Being Moderated
    I had set ulimit to unlimited, so i am not sure how much is locked by timesten.
    Is there any way to know how much memory timesten has locked?
    Is it PermSize + TempSize + LogFileSize + LogBufMB?
    It is effectively PermSize + TempSize + LogFileSize + LogBufMB _ small overhead (around 20 MB in current release). The best way to get an exact figure is to use O/S 'ipcs' command to see exact size of Timesten datastore segment.

    Chris
  • 36. Re: Perfomance Tunning required in Timesten
    ChrisJenkins Guru
    Currently Being Moderated
    About your previous message, regarding AWT write up to oracle.
    A secondary, but major, concern is this:
    Just want to understand, does writing of replication agent towards oracle impact the overall response time?
    I must be wrong but my understanding was that timesten will take writing towards oracle asynchronously for AWT cache groups.
    It will not impact my application performance.
    Sorry, but just want to understand the behavior.
    AWT cache groups are asynchronous so thgere is no direct impact on application performance. However, when replication (and AWT is included in this) enters any kind of backlog situation the replication agent has to start capturing changes from the logs on disk instead of from the in-memory log buffer (the evidence that thsi is happenning is the increasing value of LOG_FS_READS). This results in a very large increase in I/O (IOPS) on the device(s) holding the datastore's log files (it essentially changes from sequential I/O to random I/O) and this will impact the system as a whole and may especially impact TimesTen logging both of which can lead to reduced performance, increased/unpredictable response time etc.

    Chris
  • 37. Re: Perfomance Tunning required in Timesten
    user10366531 Newbie
    Currently Being Moderated
    Thank you so much Chrish, for this detail understanding.
    I have very well understood this now.

    For AWT, I am trying to set CacheAwtParallelism=4 in sys.odbc.ini

    But I am not able to connect, i am getting below error.

    6228: Invalid value (4) for CacheAwtParallelism connection attribute -- value must be the same as the current data store value (1)
  • 38. Re: Perfomance Tunning required in Timesten
    ChrisJenkins Guru
    Currently Being Moderated
    CacheAwtParallelism is a 'first connection attribute' (although the docs don't make this clear - I will log a doc bug on that). It takes effect when the datastroe is loaded into memory and cannot be changed while the datatsore is in memory. To change it you must:

    1. Stop all applications connected to the datatsore.
    2. Stop replication and cache agent.
    3. If using a non-defaulkt ramPolicy take required actions to unload datastore from memory.
    4. Use ttStatus to chCheck that the datastore is unloaded from memory
    5. Add/remove/change required attributes in sys.odbc.ini
    6. Connect to datastore as instance administrator 9to ensure changes are applied and attribute values are correct)
    7. Restart cache and replication agents
    8. Restart applications

    Note that the value specified should not exceed the value for LogBufParallelism and you also need to ensure that ReplicationApplyOrdering=0 is specified. Also, please check the IMDB Cache Guide section on parallel apply for improtant considerations you need to check before enabling parallel propagation.

    Chris
  • 39. Re: Perfomance Tunning required in Timesten
    user10366531 Newbie
    Currently Being Moderated
    Dear Chris,

    I did the same thing.

    1. ttAdmin -ramunload
    2. ttDaemonAdmin -stop
    3. ttDaemonAdmin -start
    4. made below changes into data store (sys.odbc.ini)
    ReplicationApplyOrdering=0
    ReplicationParallelism=2
    CacheAwtParallelism=4
    LogBufParallelism=8
    MemoryLock=4
    5. ttAdmin -ramload crestel_rnc_ttdb
    6. ttIsql
    7. connect dsn=<my dsn>;
    6228: Invalid value (4) for CacheAwtParallelism connection attribute -- value must be the same as the current data store value (1)
    The command failed.
  • 40. Re: Perfomance Tunning required in Timesten
    ChrisJenkins Guru
    Currently Being Moderated
    Sorry, my bad. CacheAwtParallelism (and ReplciationParallelism) are database level attributes not first connection attributes. To change them you have to drop the database (using ttDestroy) and then re-create it with the correct attribute values. Of course, if there is any data in your database that you want to preserve then you should be sure to save it first using ttMigrate, ttBulkCp or similar before destroying the database!

    Chris
  • 41. Re: Perfomance Tunning required in Timesten
    user10366531 Newbie
    Currently Being Moderated
    No Problem Chris, I will do the same.
    But just for understanding purpose, what should be the idle value for these parameters.

    I have 8 core (2 * Quad Core CPU) machine with 14 GB RAM.
  • 42. Re: Perfomance Tunning required in Timesten
    ChrisJenkins Guru
    Currently Being Moderated
    There is no easy way to predict the 'ideal' values as there are many considerations. It's an empirical exercise to figure this out.

    - If you are not using Replication then the value for ReplicationParallelism is immaterial. I know you are using AWT caching - are you also using, or planning to use, TimesTen to Timesten replication?

    - Increasing replication / AWT parallelism potentially increases the throughpout of replication and/or AWT (subject to inter-transactional dependencies in the workload) at the cost of increased CPU resource used by replication and AWT when under load. This CPU resourcewill then be unavailable for other processing.

    - AWT throughput will ultimately be limited by the backend Oracle DB depending on the hardware and software setup for that, increasing CacheAWTParallelism beyond a certainb point may not buy you anything.

    So, basically you need to consider the tradeoffs. A key factor is that you cannot afford for AWT/replication to enter a backlog state for more than a very short time and ideallly never. So, you need to do tests to determine the maximum workload that you can sustain on thsi hardware without replication/AWt backlogging and which will still give you acceptable response times (i.e. you need to try and keep the system running at no more than 60-70% utilisation). It's really about balancing all these things (including the application). You might find that you cannot achieve a balance that meets all the requirements in which case you probably need bigger hardware for Timesten, Oracle DB or both.

    If you are not using replication then I would suggest starting by just increasing CacheAWTParallelism to 2 and see what that does - be sure to monitor LOG_FS_READS throughpout the test run to be sure youdo not get an AWT backlog. If you do, try increasing CacheAWTParallelism to 4 and try again. You can go as far as 8 for CacheAWTParallelism with this hardware but I doubt that will be necessary/beneficial.

    Chris
  • 43. Re: Perfomance Tunning required in Timesten
    user10366531 Newbie
    Currently Being Moderated
    - If you are not using Replication then the value for ReplicationParallelism is immaterial. I know you are using AWT caching - are you also using, or planning to use, TimesTen to Timesten replication?

    For now we are not planning this.

    I have not set any Parallelism parameters for now.
    Just made once change which is MemoryLock=4

    I saw that the reading has really improved, but i still saw lot of si so in vmstat.
    As you suggested, we observed LOG_FS_READS and it has not increased at all.
    Please find below monitor output.

    After Application Startup


    TIME_OF_1ST_CONNECT     Thu Apr 19 11
    DS_CONNECTS     153
    DS_DISCONNECTS     26
    DS_CHECKPOINTS     3
    DS_CHECKPOINTS_FUZZY     3
    DS_COMPACTS     0
    PERM_ALLOCATED_SIZE     4196352
    PERM_IN_USE_SIZE     1079788
    PERM_IN_USE_HIGH_WATER     1079788
    TEMP_ALLOCATED_SIZE     1048576
    TEMP_IN_USE_SIZE     45561
    TEMP_IN_USE_HIGH_WATER     45561
    SYS18     0
    TPL_FETCHES     0
    TPL_EXECS     0
    CACHE_HITS     0
    PASSTHROUGH_COUNT     366
    XACT_BEGINS     18185
    XACT_COMMITS     18149
    XACT_D_COMMITS     18
    XACT_ROLLBACKS     34
    LOG_FORCES     9
    DEADLOCKS     0
    LOCK_TIMEOUTS     0
    LOCK_GRANTS_IMMED     18082
    LOCK_GRANTS_WAIT     0
    SYS19     0
    CMD_PREPARES     202
    CMD_REPREPARES     0
    CMD_TEMP_INDEXES     0
    LAST_LOG_FILE     30
    REPHOLD_LOG_FILE     25
    REPHOLD_LOG_OFF     243927400
    REP_XACT_COUNT     0
    REP_CONFLICT_COUNT     0
    REP_PEER_CONNECTIONS     3
    REP_PEER_RETRIES     0
    FIRST_LOG_FILE     25
    LOG_BYTES_TO_LOG_BUFFER     84200
    LOG_FS_READS     190020
    LOG_FS_WRITES     11
    LOG_BUFFER_WAITS     0
    CHECKPOINT_BYTES_WRITTEN     24048008
    CURSOR_OPENS     937
    CURSOR_CLOSES     937
    SYS3     0
    SYS4     0
    SYS5     0
    SYS6     0
    CHECKPOINT_BLOCKS_WRITTEN     85
    CHECKPOINT_WRITES     149
    REQUIRED_RECOVERY     0
    SYS11     0
    SYS12     1
    TYPE_MODE     0
    SYS13     0
    SYS14     0
    SYS15     186849
    SYS16     0
    SYS17     0
    SYS9     

    After 1,500,000 requests processed


    TIME_OF_1ST_CONNECT     Thu Apr 19 11
    DS_CONNECTS     343
    DS_DISCONNECTS     216
    DS_CHECKPOINTS     10
    DS_CHECKPOINTS_FUZZY     10
    DS_COMPACTS     0
    PERM_ALLOCATED_SIZE     4196352
    PERM_IN_USE_SIZE     2657837
    PERM_IN_USE_HIGH_WATER     2658136
    TEMP_ALLOCATED_SIZE     1048576
    TEMP_IN_USE_SIZE     46131
    TEMP_IN_USE_HIGH_WATER     46194
    SYS18     0
    TPL_FETCHES     0
    TPL_EXECS     0
    CACHE_HITS     0
    PASSTHROUGH_COUNT     1000365
    XACT_BEGINS     4058036
    XACT_COMMITS     4050136
    XACT_D_COMMITS     60
    XACT_ROLLBACKS     286
    LOG_FORCES     16840
    DEADLOCKS     0
    LOCK_TIMEOUTS     0
    LOCK_GRANTS_IMMED     29525281
    LOCK_GRANTS_WAIT     38
    SYS19     59
    CMD_PREPARES     237
    CMD_REPREPARES     0
    CMD_TEMP_INDEXES     0
    LAST_LOG_FILE     33
    REPHOLD_LOG_FILE     25
    REPHOLD_LOG_OFF     243927400
    REP_XACT_COUNT     2999976
    REP_CONFLICT_COUNT     0
    REP_PEER_CONNECTIONS     3
    REP_PEER_RETRIES     0
    FIRST_LOG_FILE     25
    LOG_BYTES_TO_LOG_BUFFER     3625386832
    LOG_FS_READS     190020
    LOG_FS_WRITES     104311
    LOG_BUFFER_WAITS     0
    CHECKPOINT_BYTES_WRITTEN     1332932624
    CURSOR_OPENS     5497927
    CURSOR_CLOSES     5498556
    SYS3     0
    SYS4     0
    SYS5     0
    SYS6     0
    CHECKPOINT_BLOCKS_WRITTEN     64876
    CHECKPOINT_WRITES     53097
    REQUIRED_RECOVERY     0
    SYS11     0
    SYS12     1
    TYPE_MODE     0
    SYS13     0
    SYS14     0
    SYS15     186849
    SYS16     0
    SYS17     0
    SYS9     

    After end of load test, 3,000,000 requests processed

    TIME_OF_1ST_CONNECT     Thu Apr 19 11
    DS_CONNECTS     438
    DS_DISCONNECTS     311
    DS_CHECKPOINTS     15
    DS_CHECKPOINTS_FUZZY     15
    DS_COMPACTS     0
    PERM_ALLOCATED_SIZE     4196352
    PERM_IN_USE_SIZE     2697178
    PERM_IN_USE_HIGH_WATER     2697380
    TEMP_ALLOCATED_SIZE     1048576
    TEMP_IN_USE_SIZE     46194
    TEMP_IN_USE_HIGH_WATER     46384
    SYS18     0
    TPL_FETCHES     0
    TPL_EXECS     0
    CACHE_HITS     0
    PASSTHROUGH_COUNT     1000365
    XACT_BEGINS     7091795
    XACT_COMMITS     7083815
    XACT_D_COMMITS     93
    XACT_ROLLBACKS     370
    LOG_FORCES     27999
    DEADLOCKS     0
    LOCK_TIMEOUTS     0
    LOCK_GRANTS_IMMED     51031984
    LOCK_GRANTS_WAIT     55
    SYS19     90
    CMD_PREPARES     255
    CMD_REPREPARES     0
    CMD_TEMP_INDEXES     0
    LAST_LOG_FILE     37
    REPHOLD_LOG_FILE     25
    REPHOLD_LOG_OFF     243927400
    REP_XACT_COUNT     5999954
    REP_CONFLICT_COUNT     0
    REP_PEER_CONNECTIONS     3
    REP_PEER_RETRIES     0
    FIRST_LOG_FILE     25
    LOG_BYTES_TO_LOG_BUFFER     6734582400
    LOG_FS_READS     190020
    LOG_FS_WRITES     176489
    LOG_BUFFER_WAITS     0
    CHECKPOINT_BYTES_WRITTEN     977498504
    CURSOR_OPENS     6997565
    CURSOR_CLOSES     6998186
    SYS3     0
    SYS4     0
    SYS5     0
    SYS6     0
    CHECKPOINT_BLOCKS_WRITTEN     115969
    CHECKPOINT_WRITES     104274
    REQUIRED_RECOVERY     0
    SYS11     0
    SYS12     1
    TYPE_MODE     0
    SYS13     0
    SYS14     0
    SYS15     186849
    SYS16     0
    SYS17     0
    SYS9
  • 44. Re: Perfomance Tunning required in Timesten
    ChrisJenkins Guru
    Currently Being Moderated
    By adding MemoryLock=4 you have prevented any of the memory used by the TimesTen datastore from being paged out. Given the vmstat output it seems that your system is under memory pressure when the workload is running and there is significant paging activity. By preventing any of the datastore being paged it seems you have improved things at the TimesTen level and this has improved TT response times and has also allowed replication agent to keep up with the workload. If this improvement is consistent across multiple test runs then this is good. However, given that you are still seeing significant paging activity according to the vmstat output I would suggest that the system is still not optimal and further improvements will be obtained by relieving the memory pressure (reducing memory usage elsewhere, adding more RAM).

    Note that we always recommend using MemoryLock=4 for production systems, benchmarks etc.

    Regards,

    Chris

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points