This discussion is archived
6 Replies Latest reply: May 1, 2013 9:48 AM by Srini Chavali-Oracle RSS

Heavy Swapping Occuring, 11.2.0.3 DB on Solaris

user287888 Newbie
Currently Being Moderated
I am having difficulty tracing the source of a "heavy swapping" problem with a single database. Any ideas on where to go next are greatly appreciated. I have copied/pasted the results of several settings/configs/events/trace files based on what I have looked at so far.

Oracle 11.2.0.3 database running on Solaris OS. This database supports a commercial product (called Fortify) that scans source code for security vulnerabilities. 99.8% of the time there are no users on this database and no transactions are taking place ... it should be idle but it reports constant swapping and it is very slow to startup, shutdown, apply patches, etc. It performs terribly compared to the other OLTP and heavy batch databases we run. This database starts up (very slowly ... takes approx 4 minutes to start), and after starting all its processes (roughly 9 minutes after startup) it begins reporting these heavy swapping events in the alert log.

Params:
memory_target=6G (even tried 10G to see if that made any difference)
Also tried manually setting pga_aggregate_target=5G, sga_target=5G, memoryimm_mode_without_autosga=FALSE (disabling AMM)

_______________________________________
Snippet from alert log:
WARNING: Heavy swapping observed on system in last 5 mins.
pct of memory swapped in [9.02%] pct of memory swapped out [0.00%].
Please make sure there is no memory pressure and the SGA and PGA
are configured correctly. Look at DBRM trace file for more details.
_______________________________________

_______________________________________
Snippet from dbrm trace file:
Heavy swapping observed in last 5 mins:    [pct of total memory][bytes]
total swpin [ 9.42%][6326816K], total swpout [ 0.00%][0K]
vm stats captured every 30 secs for last 5 mins:
swpin:                 swpout:
[ 1.41%][    952240K]  [ 0.00%][         0K]
[ 0.39%][    266032K]  [ 0.00%][         0K]
[ 1.46%][    981656K]  [ 0.00%][         0K]
[ 0.79%][    536192K]  [ 0.00%][         0K]
[ 0.78%][    524584K]  [ 0.00%][         0K]
[ 0.68%][    457696K]  [ 0.00%][         0K]
[ 0.76%][    510792K]  [ 0.00%][         0K]
[ 0.55%][    369312K]  [ 0.00%][         0K]
[ 0.89%][    602664K]  [ 0.00%][         0K]
[ 1.67%][   1125648K]  [ 0.00%][         0K]
Heavy swapping observed in last 5 mins:    [pct of total memory][bytes]
_______________________________________

_______________________________________
Snippet from mmon trace file (not sure what this means but it accompanies the dbrm trace file):
*** 2013-05-01 06:53:02.378
minact-scn master-status: grec-scn:0x04fa.4741d51b gmin-scn:0x04fa.4741d51b gcalc-scn:0x04fa.4741d51b

*** 2013-05-01 06:58:03.353
minact-scn master-status: grec-scn:0x04fa.4741d633 gmin-scn:0x04fa.4741d633 gcalc-scn:0x04fa.4741d633
minact-scn master-status: grec-scn:0x04fa.4741d780 gmin-scn:0x04fa.4741d780 gcalc-scn:0x04fa.4741d780

*** 2013-05-01 07:08:08.480
minact-scn master-status: grec-scn:0x04fa.4741d831 gmin-scn:0x04fa.4741d831 gcalc-scn:0x04fa.4741d831

*** 2013-05-01 07:13:12.356
minact-scn master-status: grec-scn:0x04fa.4741dace gmin-scn:0x04fa.4741dace gcalc-scn:0x04fa.4741dace
_______________________________________

_______________________________________
Results of: SHOW PARAMETER SERVERS;
NAME                            TYPE     VALUE
max_shared_servers          integer
parallel_max_servers          integer   50
parallel_min_servers           integer   8
parallel_servers_target       integer   50
shared_servers                 integer   1
_______________________________________

_______________________________________
Results of: vmstat
kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s0 s1 s2 --   in   sy   cs us sy id
 0 0 0 5309392 15835712 514 2048 14959 0 0 0 0 5 5 -21 0 4606 10083 5281 1 2 98
_______________________________________

_______________________________________
Results of: top (doesn't change a whole lot)
load averages:  1.12,  1.19,  1.20                                               07:08:04
96 processes:  95 sleeping, 1 on cpu
CPU states: 97.5% idle,  0.7% user,  1.8% kernel,  0.0% iowait,  0.0% swap
Memory: 64G real, 15G free, 26G swap in use, 3277M swap free

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
  4734 root      67  59    0   61M   56M sleep   23.5H  0.04% cvm
  4979 root      18  59    0   46M   21M sleep   21.2H  0.03% cma
 16365 root       1  38    0 2472K 1824K sleep    0:00  0.02% sadc
  9501 oracle     1  59    0    0K    0K sleep    0:14  0.01% oracle
 23838 oracle     3  59    0    0K    0K sleep  101:25  0.01% tnslsnr
  9297 oracle     1  59    0    0K    0K sleep    0:13  0.01% oracle
  9361 oracle     1  59    0    0K    0K sleep    0:12  0.01% oracle
 27236 oracle     1  59    0    0K    0K sleep    0:08  0.01% oracle
  9476 oracle    11  59    0    0K    0K sleep    0:07  0.01% oracle
 10384 oracle     1  59    0    0K    0K sleep    0:11  0.01% oracle
  9495 oracle     1  59    0    0K    0K sleep    0:10  0.01% oracle
  9447 oracle    11  59    0    0K    0K sleep    0:01  0.01% oracle
 17987 fortify   76  59    0 1714M  812M sleep  149:57  0.00% java
 24567 oracle     7  59    0  173M  158M sleep  148:45  0.00% emagent
  9214 oracle     1  59    0    0K    0K sleep    0:06  0.00% oracle
_______________________________________

_______________________________________
Finally, the results of /usr/sbin/swap -l:
swapfile             dev  swaplo blocks   free
/dev/swap           4294967295,4294967295     16 33554416 33554416
_______________________________________

Edited by: 825118 on May 1, 2013 9:51 AM
  • 1. Re: Heavy Swapping Occuring, 11.2.0.3 DB on Solaris
    saratpvv Journeyer
    Currently Being Moderated
    Have a look at note 1452790.1.

    https://kr.forums.oracle.com/forums/thread.jspa?threadID=2441645
  • 2. Re: Heavy Swapping Occuring, 11.2.0.3 DB on Solaris
    user287888 Newbie
    Currently Being Moderated
    Thank you for the quick reply but I am not sure that applies this database was/is a clean installed 11.2.0.3 DB. From what I can tell based on that thread (which I have seen before during my research of this problem), it appears as though the people replying don't consider it to be actually swapping ... more like erroneous reporting of swapping. Please feel free to elaborate more if I am missing something as I am running out of ideas. Thanks again.
  • 3. Re: Heavy Swapping Occuring, 11.2.0.3 DB on Solaris
    Srini Chavali-Oracle Oracle ACE Director
    Currently Being Moderated
    Pl post the exact OS version - I assume all of the pre-reqs for the install were met ?

    http://docs.oracle.com/cd/E11882_01/install.112/e24346/toc.htm

    Oracle Database (RDBMS) on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2) [ID 169706.1]

    Requirements To Install Oracle 11gR2 RDBMS On Solaris 11 SPARC / X86_64 [ID 1496618.1]

    Have you opened an SR with Support ? You might also want to post this in the Solaris forums to see if someone there can help

    HTH
    Srini
  • 4. Re: Heavy Swapping Occuring, 11.2.0.3 DB on Solaris
    user287888 Newbie
    Currently Being Moderated
    Thanks for the suggestion to post on the Solaris forum ... I just posted it (with the exact OS version).

    OS version: Solaris OS 10 Update 9/SunOS 5.10 Generic_147440-27 sun4u sparc SUNW kernel.

    Yes, all pre-reqs were met for the install. Our OS is a "standard" install we use across VMs. Our Oracle install was actually performed by the software vendor with local DBA support. Of course, with all that said, there is always the possibility something was set up incorrectly ... which is what I am hoping to track down.

    Edited by: 825118 on May 1, 2013 9:36 AM
  • 5. Re: Heavy Swapping Occuring, 11.2.0.3 DB on Solaris
    sb92075 Guru
    Currently Being Moderated
    To see how to properly format results

    How do I ask a question on the forums?
    SQL and PL/SQL FAQ


    scroll down to #9 to see how to use
     tags
    
    post formatted results from OS command like below
    
    [code]
    [oracle@localhost ~]$ vmstat 6 10
    procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     0  0 109004 202720  32248 385396    1    3    18    30  198  291  1  2 97  1  0
     0  0 109004 202596  32264 385388    0    0     0    36 1054 1278  1  1 98  1  0
     1  0 109004 202620  32280 385388    0    0     0    20 1000 1192  1  2 98  0  0
     0  0 109004 202612  32288 385388    0    0     0    13  999 1225  1  1 99  0  0
     0  0 109004 211176  32304 385388    0    0     1    19 1002 1196  2  4 95  0  0
     0  0 109004 210392  32320 385428    0    0     0    49 1090 1345  1  5 94  0  0
     0  0 109004 210368  32328 385428    0    0     0    25 1087 1330  2  2 96  0  0
     0  0 109004 210540  32344 385428    0    0     0    15 1001 1195  1  2 98  0  0
     0  0 109004 210556  32360 385420    0    0     0    32 1002 1212  1  1 98  0  0
     1  0 109004 210408  32368 385440    0    0     1    20 1002 1188  1  2 97  0  0
    [oracle@localhost ~]$ 
    [/code]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
  • 6. Re: Heavy Swapping Occuring, 11.2.0.3 DB on Solaris
    Srini Chavali-Oracle Oracle ACE Director
    Currently Being Moderated
    Pl see if this MOS Doc is relevant

    Solaris Server Using Excessive Swap Space When There is Plenty of Free RAM [ID 761960.1]

    HTH
    Srini

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points