9 Replies Latest reply: Nov 24, 2010 4:44 AM by chris_here RSS

    ADDM finding "virtual memory paging"

    chris_here
      Using 11.2.0.2 on Centos 5.4 x86_64, I get that entry all the time as top ADDM finding with 100% impact, whatever that means:

      "Significant virtual memory paging was detected on the host operating system"

      I'm unable to identify (eg with top) a process that could create significant swapping, or even consume any significant amount of memory. The box has 50G of RAM, 40G being allocated to the DB, managed using Automatic Memory Management. Current system state:
      %free -m
                   total       used       free     shared    buffers     cached
      Mem:         52335      40957      11377          0       1383      27084
      -/+ buffers/cache:      12489      39845
      Swap:        32767       7689      25078
      
      %vmstat
      procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
       r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       3  0 7875588 11631444 1416968 27729992    0    0    26    45    1    0  2  1 97  0  0
      The "Pages Paged-out" metric indeed shows a few peaks at about 2000 pages per sec sometimes, but they are few and far between; moreover I'm unable to associate them with any particular cause.

      Any idea what could be the cause, or how to indestigate this further?

      Your help would be appreciated.
      Chris
        • 1. Re: ADDM finding "virtual memory paging"
          Oviwan
          post output of "df -h". are there any other application running on that system? normally a database should never swap, also oracle recommends a swap space of 16GB if you have >16GB memory. are any errors in the alert log?
          • 2. Re: ADDM finding "virtual memory paging"
            chris_here
            Hi,

            df gives this:
            %df -h
            Filesystem            Size  Used Avail Use% Mounted on
            /dev/mapper/Raid5-Sys
                                   20G  2.6G   16G  14% /
            /dev/mapper/Raid5-Home
                                  291G   33G  243G  12% /home
            /dev/cciss/c0d0p1      99M   39M   55M  42% /boot
            tmpfs                  44G   27G   18G  61% /dev/shm
            The system is pretty much idle, this is a test system with some apps but no real workload. It has currently 32G of swap space, the recommendations in this regard are somewhat contradictory, the automatic recommendations on My Support where pushing for a swap space equal to memory.

            Besides, no recent alert in the logs.

            Thanks for your help,
            Chris

            Edited by: chris3110 on Nov 20, 2010 12:03 AM
            • 3. Re: ADDM finding "virtual memory paging"
              sb92075
              take 2 minutes and post results as shown below
              bcm@bcm-laptop:~$ vmstat 6 20
              procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
               r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
               0  0      0 206984 176564 2464900    0    0    17    40  418  331 11  4 85  1
               0  0      0 207272 176568 2464900    0    0     0    55 1408 3318 12  6 81  0
               4  0      0 207476 176576 2464900    0    0     0    15 1095 2514  7  5 88  0
               0  0      0 208476 176584 2464900    0    0     0    58 1001 2516  8  4 88  0
               1  0      0 213764 176592 2464900    0    0     0    25 1071 2546  7  4 88  0
               1  0      0 213952 176596 2464900    0    0     0   347 1178 2669  8  5 85  1
               0  0      0 213796 176612 2464900    0    0     0   106  968 2431 16  4 79  1
               3  0      0 210448 176624 2464900    0    0     0   108 1922 4531 19 11 69  1
               1  0      0 199628 176628 2464944    0    0     0    50 1396 3319 14  6 80  0
               0  0      0 205720 176636 2464856    0    0     0   107 1180 2858 14  6 80  0
               0  0      0 205720 176640 2464856    0    0     0    23 1088 2699  7  5 88  0
               0  0      0 205348 176644 2464856    0    0     0   161 1127 2789 13  5 82  1
               0  0      0 205100 176652 2464856    0    0     0   111 1157 2752 11  4 84  1
               0  0      0 204976 176660 2464860    0    0     0    67 1344 3292 11  8 81  0
               1  0      0 208712 176676 2464860    0    0     0   289 1071 2645 13  5 83  0
               1  0      0 213424 176676 2464860    0    0     0    21 1035 2588  8  4 88  0
               0  0      0 215036 176680 2464860    0    0     0    45 1102 2569  9  6 85  0
               1  0      0 215036 176688 2464876    0    0     0    37 1119 2587  7  5 87  0
               0  0      0 214912 176688 2464860    0    0     0    82 1161 2569  7  4 89  0
               0  0      0 207996 176692 2464860    0    0     0    85 1260 2724 11  5 83  0
              bcm@bcm-laptop:~$ 
              • 4. Re: ADDM finding "virtual memory paging"
                chris_here
                Here it is, under some realistic workload.
                # vmstat 6 20
                procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
                 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
                 0  0 7139728 9657636 1417644 30140672    0    0    26    45    1    0  2  1 97  0  0
                 1  0 7139728 9658544 1417644 30140688    0    0    43   447 1166 9059  1  0 99  0  0
                 2  0 7139728 9626756 1417644 30141120    0    0    11   144 1284 9393  3  0 97  0  0
                 0  0 7139236 9630452 1417644 30144836  110    0   156  1751 1816 10719  3  1 94  2  0
                 1  0 7138888 9623444 1417644 30145192    2    0    13  1471 1099 9764  1  1 98  0  0
                 0  0 7138104 9620356 1417644 30148064  160    0  1476   681 1969 10154  3  1 95  1  0
                 1  0 7137984 9617476 1417644 30148204    6    0    27   804 2490 9980  1  0 98  0  0
                 1  0 7137808 9640948 1417648 30148432   27    0    81   283 1455 10105  1  0 98  0  0
                 0  0 7137664 9644652 1417648 30148600    5    0    51   296 1126 9891  1  1 98  0  0
                 1  0 7137664 9641720 1417648 30148612    0    0    56    99 1083 9633  1  1 98  0  0
                 0  0 7136912 9643780 1417648 30149540  105    0   289   386 1402 9220  2  0 98  0  0
                 0  0 7136912 9646024 1417648 30149552    0    0    64   445 1074 9514  1  0 99  0  0
                 1  0 7136912 9610620 1417648 30149932    0    0    37   171 1501 10268  3  0 97  0  0
                 0  0 7136200 9614500 1417648 30151220  109    0   410  1334 1328 10001  2  0 97  1  0
                 1  0 7136152 9606144 1417648 30151244    7    0    18  1272 1180 10168  1  1 98  0  0
                 0  0 7136152 9607596 1417652 30152076    0    0   105   742 1524 9695  4  1 95  0  0
                 1  0 7135828 9608328 1417652 30152420   13    0    40   977 1167 9427  1  0 99  0  0
                 1  0 7135796 9608120 1417652 30152568    6    0    22   238 1397 9357  1  0 98  0  0
                 0  0 7135764 9613560 1417652 30152692    1    0    57   253 2075 9924  1  1 98  0  0
                 1  0 7135764 9608776 1417652 30152700    0    0    45   272 1106 9752  1  1 98  0  0
                And EM "Paging Activity" metric gives this:
                Active Pages     6813331
                Inactive Dirty Pages     
                Inactive Clean Pages     
                Pages Paged-out (per second)     621.17
                Pages Paged-in (per second)     230.62
                Pages Scanned by Page Stealing Daemon (per second)     0.37
                Chris
                • 5. Re: ADDM finding "virtual memory paging"
                  sb92075
                  paging activity is different from swapping.
                  paging is normal & expected
                  Your system is doing minor swapping (si & so) but it is nothing I'd loose any sleep over.

                  please post formatted results from SQL as shown below
                  SQL> select * from V_$SGA_TARGET_ADVICE;
                  
                    SGA_SIZE SGA_SIZE_FACTOR ESTD_DB_TIME ESTD_DB_TIME_FACTOR ESTD_PHYSICAL_READS
                  ---------- --------------- ------------ ------------------- -------------------
                         488          .5         429           1.4342           267989
                         732            .75         299             1           256719
                         976           1         299             1           256719
                        1220           1.25         299             1           256719
                        1464            1.5         299             1           256719
                        1708           1.75         299             1           256719
                        1952           2         299             1           256719
                  
                  7 rows selected.
                  • 6. Re: ADDM finding "virtual memory paging"
                    chris_here
                    Here it is (under idle load in case it makes any difference):
                    %select * from V_$SGA_TARGET_ADVICE;
                          6144             .25       951839                1.01             4700087
                          9216            .375       949012               1.007             4288159
                         12288              .5       948164              1.0061             4133084
                         15360            .625       945619              1.0034             3681023
                         18432             .75       943640              1.0013             3405548
                         21504            .875       942698              1.0003             3248226
                         24576               1       942415                   1             3210661
                         27648           1.125       942227               .9998             3177591
                         30720            1.25       942038               .9996             3155759
                         33792           1.375       941944               .9995             3150622
                         36864             1.5       941850               .9994             3118194
                         39936           1.625       941755               .9993             3111773
                         43008            1.75       941661               .9992             3088014
                         46080           1.875       941567               .9991             3088014
                         49152               2       941567               .9991             3088014
                    
                    15 rows selected.
                    By the way the "Swap Utilization" metrics has been consistently hovering around 20% during the last 7 days, max value = 24%
                    • 7. Re: ADDM finding "virtual memory paging"
                      sb92075
                      By the way the "Swap Utilization" metrics has been consistently hovering around 20% during the last 7 days, max value = 24%
                      SWAP space exists so that in case it is needed, it can be utilized.
                      The fact that some swap space is "used" is not an immediate indicator that RAM is in short supply or that RAM is a bottleneck.
                      If a process gets swapped out, then simply stay in swap space why is this a bad thing?
                      A process stays in swap space as long as it stays idle.

                      What is a bad thing is swapping ACTIVITY (swap in & swap out RATES).

                      If I were you, I'd reduce SGA to be about 25% smaller than it is now.

                      Afterwards, monitor what happens to swapping activity.
                      • 8. Re: ADDM finding "virtual memory paging"
                        chris_here
                        Ok I'll try that and keep you informed.

                        Thanks for the help,
                        Chris
                        • 9. Re: ADDM finding "virtual memory paging"
                          chris_here
                          Hi sb,

                          I've disabled AMM and set SGA and PGA to 35% of total memory as advised in the documentation for OLAP activity. This roughly corresponds to the figure you advised too. However I still get "Virtual Memory Paging with 100% impact, 21/23 occurrences last 24h", with ADDM all the time. The database is not heavilly loaded and I'm not able to identify any suspicious or disquietening avtivity with the system (memory utilization, swap utilization, etc). vmstat mostly indicates 0 pi and po.

                          It does look to me as a false indication although I'm not able to tell for sure. I've read that this happens under Windows but not under Linux, which is the case at hand.

                          Any idea?

                          Thanks for your help,
                          Chris