This discussion is archived
1 2 Previous Next 28 Replies Latest reply: Jul 22, 2013 8:51 PM by user12024958 Go to original post RSS
  • 15. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated
    BTW the defunct processes from the oracle BG processes stay the same set of pids, ie. termined childs properly, from startup, that just seems never to be picked up by their parent (waitpid(2))
  • 16. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated
    user5287726 running your hot kernel count f.ex. about a minute or two I see this:
    #root:~# cat scripts/busy_kernel_counts.d
    #!/usr/sbin/dtrace -s
     
    #pragma D option quiet
    
    dtrace:::BEGIN
    {
         printf( "Started...\n" );
    }
     
    profile:::profile-1001hz
    / arg0 / 
    {
        @hot[ arg0 ] = count();
    }
     
    dtrace:::END
    {
        printa( "%@u %a\n", @hot );
    }
    
    #root:~# scripts/busy_kernel_counts.d
    Started...
    <snip>
    1139 unix`current_thread+0x290
    1165 unix`lock_set_spl+0x14
    1354 unix`mutex_exit+0x8
    1366 unix`shipit+0x28
    1426 unix`disp_anywork+0xcc
    1445 unix`_resume_from_idle+0xd0
    1872 unix`disp_getwork+0x1a4
    2197 unix`atomic_add_64+0x18
    2260 unix`disp_getwork+0x194
    2273 SPARC-T4`cpu_inv_tsb+0x10
    2858 unix`disp_anywork+0xbc
    2882 unix`disp_getwork+0x180
    3375 unix`drv_usecwait+0xa8
    4387 unix`mutex_enter+0x8
    30174594 unix`cpu_halt+0x158
    cpu_halt +30M would this imply cpu preemption is too high and thus making contention var busy, possible stalling access?

    Should we possible think of:

    1 - trying another cpu scheduler, aka fair scheduler versus time share
    2 - change time slice for TS scheduler

    I would just expect Solaris 11.1 on std. SPARC to perform better, so I'm more likely to think that the way our DBAs got their 11g + data guard installed/configured is driving HW+OS wrong/bad, if it's not a bug somehow.
  • 17. Re: simple processes are very slow at intervals
    800381 Explorer
    Currently Being Moderated
    That's during one of your slowdowns? I've never seen that one before. Ugh.

    Did you change the sar frequency to something often? If so, what does that show during the slowdowns?
  • 18. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated
    'What' is you mean during slow down?

    No, sorry didn't went the SAR route yet as I profiled with prstat+Dtraces

    Slow downs can be hard to capture as they are spiky at times, but yes I find many condition var of unix`_resume_from_idle in even simple cli cmdx during slow downs.

    At one point I glimpsed +100k ctx sw/s with top, which I find high with runqsz in the 1x range and finding ora_vmtm_SIDx processes running at 1ms precision making 1k ctx sw/s, dunno it that's alarming, thought it should run every 10ms, but G finds it varies 1, 4, 10, 20ms guess it depends on your HW resources maybe. Hard to figure out the culprit cause :(
  • 19. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated
    Looking into which processes are doing a lot of ctx sw it seems that

    - ora_vktm_<SID> as they run every 1ms when solaris 11.1 seems to def highres_tick = 1
    - ZFS system daemons periodically are doing many ctx sw, properly when flushing ARC?

    Sampling ZFS ctx sw under moderate app load from the current busiest SID's 3x spools, is this wildly many?
     
    #root:~# prstat -v -p `ps -ef|awk 'BEGIN{i=0} !/awk / && /zpool-SID1/{pid=$2;i++} END{for (i--; i>0; i--) printf pid[i]",";printf pid[0]"\n"}'` -c 5 30
    Please wait...
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 126 0 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 3 0 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2 0 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.25, 5.35, 5.30
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 32K 75 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 766 19 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 688 2 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.21, 5.34, 5.30
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 29K 72 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 3 0 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 685 3 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.17, 5.33, 5.30
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 26K 79 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 3 0 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 495 2 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.12, 5.32, 5.29
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 31K 66 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 532 10 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 616 0 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.10, 5.31, 5.29
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 126 0 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 588 10 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 624 0 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.07, 5.30, 5.29
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 126 0 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 640 15 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 534 7 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.05, 5.29, 5.29
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 126 0 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 678 7 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 591 3 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.05, 5.29, 5.29
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 126 0 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 649 6 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 560 1 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.04, 5.29, 5.28
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 31K 80 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 529 2 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 586 1 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.01, 5.27, 5.28
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.2 0.0 0.0 0.0 0.0 100 0.0 44K 104 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 646 9 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 680 0 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.01, 5.27, 5.28
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.2 0.0 0.0 0.0 0.0 100 0.0 36K 83 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 597 18 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2 0 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.01, 5.27, 5.27
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 35K 111 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 683 9 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2 0 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.00, 5.26, 5.27
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.2 0.0 0.0 0.0 0.0 100 0.0 35K 111 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 558 5 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 603 5 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.02, 5.26, 5.27
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 34K 115 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 607 12 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 565 7 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.01, 5.25, 5.27
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 37K 77 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 712 5 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 575 3 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 4.99, 5.25, 5.27
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.2 0.0 0.0 0.0 0.0 100 0.0 38K 88 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 612 8 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 559 3 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 4.98, 5.24, 5.27
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 126 0 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 636 2 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 551 1 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 4.96, 5.23, 5.26
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    928 root 0.0 0.7 0.0 0.0 0.0 0.0 99 0.0 1K 464 0 0 zpool-SID1arc/262
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 126 0 0 0 zpool-SID1/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 2K 25 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 4.94, 5.22, 5.26
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 31K 73 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 4K 17 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 693 6 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.02, 5.23, 5.27
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 35K 96 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 846 11 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 630 7 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.10, 5.25, 5.27
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 34K 104 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 885 4 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 579 2 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.06, 5.24, 5.27
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 34K 86 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 898 12 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 695 1 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.06, 5.23, 5.26
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 37K 86 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 837 11 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 700 4 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.08, 5.23, 5.26
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 37K 96 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 774 12 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 567 1 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.06, 5.23, 5.26
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 126 0 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 702 18 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 677 2 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.04, 5.22, 5.26
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 126 0 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 655 2 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 524 0 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 5.02, 5.21, 5.25
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.2 0.0 0.0 0.0 0.0 100 0.0 37K 80 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 666 8 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 644 1 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 4.98, 5.20, 5.25
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.2 0.0 0.0 0.0 0.0 100 0.0 42K 111 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 687 10 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 525 0 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 4.98, 5.20, 5.25
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    852 root 0.0 0.2 0.0 0.0 0.0 0.0 100 0.0 44K 128 0 0 zpool-SID1/262
    928 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 3 0 0 0 zpool-SID1arc/262
    930 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 593 0 0 0 zpool-SID1red/262
    Total: 3 processes, 786 lwps, load averages: 4.98, 5.20, 5.25
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
  • 20. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated
    Just FYI

    # Top ZFS settings of my SID's three pools
    #root:~# zfs get all SID1
    NAME  PROPERTY              VALUE                  SOURCE
    SID1   aclinherit            restricted             default
    SID1   aclmode               discard                default
    SID1   atime                 off                    local
    SID1   available             2.60T                  -
    SID1   canmount              on                     default
    SID1   casesensitivity       mixed                  -
    SID1   checksum              on                     default
    SID1   compression           off                    default
    SID1   compressratio         1.00x                  -
    SID1   copies                1                      default
    SID1   creation              Mon Apr 22 14:08 2013  -
    SID1   dedup                 off                    default
    SID1   devices               on                     default
    SID1   encryption            off                    -
    SID1   exec                  on                     default
    SID1   keychangedate         -                      default
    SID1   keysource             none                   default
    SID1   keystatus             none                   -
    SID1   logbias               latency                default
    SID1   mlslabel              none                   -
    SID1   mounted               yes                    -
    SID1   mountpoint            /global/SID1            local
    SID1   multilevel            off                    -
    SID1   nbmand                off                    default
    SID1   normalization         none                   -
    SID1   primarycache          all                    default
    SID1   quota                 none                   default
    SID1   readonly              off                    default
    SID1   recordsize            128K                   default
    SID1   referenced            39K                    -
    SID1   refquota              none                   default
    SID1   refreservation        none                   default
    SID1   rekeydate             -                      default
    SID1   reservation           none                   default
    SID1   rstchown              on                     default
    SID1   secondarycache        all                    default
    SID1   setuid                on                     default
    SID1   shadow                none                   -
    SID1   share.*               ...                    local
    SID1   snapdir               hidden                 default
    SID1   sync                  standard               default
    SID1   type                  filesystem             -
    SID1   used                  1.70T                  -
    SID1   usedbychildren        1.70T                  -
    SID1   usedbydataset         39K                    -
    SID1   usedbyrefreservation  0                      -
    SID1   usedbysnapshots       0                      -
    SID1   utf8only              off                    -
    SID1   version               6                      -
    SID1   vscan                 off                    default
    SID1   xattr                 on                     default
    SID1   zoned                 off                    default
    
    #root:~# zfs get all SID1arch
    NAME     PROPERTY              VALUE                  SOURCE
    SID1arch  aclinherit            restricted             default
    SID1arch  aclmode               discard                default
    SID1arch  atime                 off                    local
    SID1arch  available             179G                   -
    SID1arch  canmount              on                     default
    SID1arch  casesensitivity       mixed                  -
    SID1arch  checksum              on                     default
    SID1arch  compression           on                     local
    SID1arch  compressratio         1.97x                  -
    SID1arch  copies                1                      default
    SID1arch  creation              Mon Apr 22 14:07 2013  -
    SID1arch  dedup                 off                    default
    SID1arch  devices               on                     default
    SID1arch  encryption            off                    -
    SID1arch  exec                  on                     default
    SID1arch  keychangedate         -                      default
    SID1arch  keysource             none                   default
    SID1arch  keystatus             none                   -
    SID1arch  logbias               throughput             local
    SID1arch  mlslabel              none                   -
    SID1arch  mounted               yes                    -
    SID1arch  mountpoint            /global/SID1/oraarch    local
    SID1arch  multilevel            off                    -
    SID1arch  nbmand                off                    default
    SID1arch  normalization         none                   -
    SID1arch  primarycache          metadata               local
    SID1arch  quota                 300G                   local
    SID1arch  readonly              off                    default
    SID1arch  recordsize            128K                   default
    SID1arch  referenced            120G                   -
    SID1arch  refquota              none                   default
    SID1arch  refreservation        none                   default
    SID1arch  rekeydate             -                      default
    SID1arch  reservation           none                   default
    SID1arch  rstchown              on                     default
    SID1arch  secondarycache        all                    default
    SID1arch  setuid                on                     default
    SID1arch  shadow                none                   -
    SID1arch  share.*               ...                    local
    SID1arch  snapdir               hidden                 default
    SID1arch  sync                  standard               default
    SID1arch  type                  filesystem             -
    SID1arch  used                  121G                   -
    SID1arch  usedbychildren        847M                   -
    SID1arch  usedbydataset         120G                   -
    SID1arch  usedbyrefreservation  0                      -
    SID1arch  usedbysnapshots       0                      -
    SID1arch  utf8only              off                    -
    SID1arch  version               6                      -
    SID1arch  vscan                 off                    default
    SID1arch  xattr                 on                     default
    SID1arch  zoned                 off                    default
    
    #root:~# zfs get all SID1redo
    NAME     PROPERTY              VALUE                  SOURCE
    SID1redo  aclinherit            restricted             default
    SID1redo  aclmode               discard                default
    SID1redo  atime                 off                    local
    SID1redo  available             8.49G                  -
    SID1redo  canmount              on                     default
    SID1redo  casesensitivity       mixed                  -
    SID1redo  checksum              on                     default
    SID1redo  compression           off                    default
    SID1redo  compressratio         1.00x                  -
    SID1redo  copies                1                      default
    SID1redo  creation              Mon Apr 22 14:06 2013  -
    SID1redo  dedup                 off                    default
    SID1redo  devices               on                     default
    SID1redo  encryption            off                    -
    SID1redo  exec                  on                     default
    SID1redo  keychangedate         -                      default
    SID1redo  keysource             none                   default
    SID1redo  keystatus             none                   -
    SID1redo  logbias               latency                local
    SID1redo  mlslabel              none                   -
    SID1redo  mounted               yes                    -
    SID1redo  mountpoint            /global/SID1/oraredo01  local
    SID1redo  multilevel            off                    -
    SID1redo  nbmand                off                    default
    SID1redo  normalization         none                   -
    SID1redo  primarycache          all                    local
    SID1redo  quota                 10G                    local
    SID1redo  readonly              off                    default
    SID1redo  recordsize            128K                   default
    SID1redo  referenced            1.51G                  -
    SID1redo  refquota              none                   default
    SID1redo  refreservation        none                   default
    SID1redo  rekeydate             -                      default
    SID1redo  reservation           none                   default
    SID1redo  rstchown              on                     default
    SID1redo  secondarycache        all                    default
    SID1redo  setuid                on                     default
    SID1redo  shadow                none                   -
    SID1redo  share.*               ...                    local
    SID1redo  snapdir               hidden                 default
    SID1redo  sync                  standard               default
    SID1redo  type                  filesystem             -
    SID1redo  used                  1.51G                  -
    SID1redo  usedbychildren        672K                   -
    SID1redo  usedbydataset         1.51G                  -
    SID1redo  usedbyrefreservation  0                      -
    SID1redo  usedbysnapshots       0                      -
    SID1redo  utf8only              off                    -
    SID1redo  version               6                      -
    SID1redo  vscan                 off                    default
    SID1redo  xattr                 on                     default
    SID1redo  zoned                 off                    default
    #ZFS childs overwriting settings:
    #root:~# zfs list -r SID1
    NAME            USED  AVAIL  REFER  MOUNTPOINT
    SID1            1.70T  2.60T    39K  /global/SID1
    SID1/data       1011G   389G  1011G  /global/SID1/oradata
    SID1/hotbackup   658G  1.31T   658G  /global/SID1/orahotbackup
    SID1/temp       13.7G  86.3G  13.7G  /global/SID1/oratemp
    SID1/undo       60.0G  40.0G  60.0G  /global/SID1/oraundo
    #root:~# zfs list -r SID1arch
    NAME            USED  AVAIL  REFER  MOUNTPOINT
    SID1arch         121G   179G   121G  /global/SID1/oraarch
    SID1arch/redo02  804M  9.22G   804M  /global/SID1/oraredo02
    #root:~# zfs list -r SID1redo
    NAME      USED  AVAIL  REFER  MOUNTPOINT
    SID1redo  1.51G  8.49G  1.51G  /global/SID1/oraredo01
    #root:~# zfs get all SID1/data | grep local
    SID1/data  logbias               throughput             local
    SID1/data  mountpoint            /global/SID1/oradata    local
    SID1/data  primarycache          all                    local
    SID1/data  quota                 1.37T                  local
    SID1/data  recordsize            8K                     local
    SID1/data  share.*               ...                    local
    #root:~# zfs get all SID1/hotbackup | grep local
    SID1/hotbackup  mountpoint            /global/SID1/orahotbackup  local
    SID1/hotbackup  quota                 1.95T                     local
    SID1/hotbackup  share.*               ...                       local
    root:~# zfs get all SID1/temp | grep local
    SID1/temp  logbias               throughput             local
    SID1/temp  mountpoint            /global/SID1/oratemp    local
    SID1/temp  primarycache          all                    local
    SID1/temp  quota                 100G                   local
    SID1/temp  share.*               ...                    local
    #root:~# zfs get all SID1/undo | grep local
    SID1/undo  logbias               throughput             local
    SID1/undo  mountpoint            /global/SID1/oraundo    local
    SID1/undo  primarycache          all                    local
    SID1/undo  quota                 100G                   local
    SID1/undo  share.*               ...                    local
    #root:~# zfs get all SID1arch/redo02 | grep local
    SID1arch/redo02  logbias               latency                local
    SID1arch/redo02  mountpoint            /global/SID1/oraredo02  local
    SID1arch/redo02  primarycache          all                    local
    SID1arch/redo02  quota                 10G                    local
    SID1arch/redo02  share.*               ...                    local
    #root:~# zfs get all SID1redo | grep local
    SID1redo  atime                 off                    local
    SID1redo  logbias               latency                local
    SID1redo  mountpoint            /global/SID1/oraredo01  local
    SID1redo  primarycache          all                    local
    SID1redo  quota                 10G                    local
    SID1redo  share.*               ...                    local
  • 21. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated
    Built our pools trying to follow ZFS BCP for Oracle DB (http://docs.oracle.com/cd/E26502_01/html/E29022/chapterzfs-db1.html#chapterzfs-db2)

    So zpools is built from minor 100-200GB LUNs of a SAN storage array like below.

    Would we possible benefict from tunning on ZFS device IO queue depth as well as IO throttling (in kernel and HBA driver)?

    http://docs.oracle.com/cd/E26502_01/html/E29022/chapterzfs-5.html#scrolltoc

    plus considering this from ZFS BCP for Oracle DB:
    Consider tuning storage array I/O queues (for systems with HDS or EMC storage arrays)
    
    ZFS aggregates read and write I/O and manages the priority of I/O before sending it to the driver level, which handles the device. The zfs_vdev_max_pending parameter defines the maximum number of I/Os that ZFS sends to any storage pool device.
    
    In a legacy storage environment, the ssd_max_throttle and sd_max_throttle parameters define the maximum number of concurrent I/Os that the driver can send to the storage. By setting the zfs_vdev_max_pending default value equal to the value of the [s]sd_max_throttle parameter, we prevent ZFS from queuing I/O to yet another unnecessary SD layer.
    
    If you have ssd:ssd_max_throttle or sd:sd_max_throttle in the /etc/system file in your existing environment, then set zfs:zfs_vdev_max_pending at the same value. For example, if the storage array administrator asked for the following setting:
    
    set ssd:ssd_max_throttle=20 
    Then, also set this parameter as follows:
    
    set ssd:ssd_max_throttle=20 
    set zfs:zfs_vdev_max_pending=20 
    Setting this parameter allows ZFS to control each LUN queue. This means that the total number of pending I/Os in the storage can grow as follows:
    
    number of LUNs * ZFS_VDEV_MAX_PENDING
    QLC driver throttle execution at default in /kernel/drv/qlc.conf:

    #Name: Execution throttle
    #Type: Integer, commands; Range: 1 - 65535; Default: 32
    #Usage: This field specifies the maximum number of commands sent per LUN
    # by the firmware.
    #NOTE: Exceeding device capabilities causes unneeded command retries
    # that impact performance.
    execution-throttle=32;


    Is it possible that tunning ZFS, kernel, HBA driver IO depth might lower ZFS daemon context switching and give better IO utilization?


    #root:~# zpool status SID1
      pool: SID1
     state: ONLINE
      scan: none requested
    config:
    
         NAME                                     STATE     READ WRITE CKSUM
         SID1                                     ONLINE       0     0     0
           c0t60050768028109972800000000000017d0  ONLINE       0     0     0
           c0t60050768028109972800000000000018d0  ONLINE       0     0     0
           c0t60050768028109972800000000000019d0  ONLINE       0     0     0
           c0t6005076802810997280000000000001Ad0  ONLINE       0     0     0
           c0t6005076802810997280000000000001Bd0  ONLINE       0     0     0
           c0t6005076802810997280000000000001Cd0  ONLINE       0     0     0
           c0t6005076802810997280000000000001Dd0  ONLINE       0     0     0
           c0t6005076802810997280000000000001Ed0  ONLINE       0     0     0
           c0t6005076802810997280000000000001Fd0  ONLINE       0     0     0
           c0t60050768028109972800000000000020d0  ONLINE       0     0     0
           c0t60050768028109972800000000000021d0  ONLINE       0     0     0
           c0t60050768028109972800000000000022d0  ONLINE       0     0     0
           c0t60050768028109972800000000000023d0  ONLINE       0     0     0
           c0t60050768028109972800000000000024d0  ONLINE       0     0     0
           c0t60050768028109972800000000000025d0  ONLINE       0     0     0
           c0t60050768028109972800000000000026d0  ONLINE       0     0     0
           c0t60050768028109972800000000000027d0  ONLINE       0     0     0
           c0t60050768028109972800000000000028d0  ONLINE       0     0     0
           c0t60050768028109972800000000000029d0  ONLINE       0     0     0
           c0t6005076802810997280000000000002Ad0  ONLINE       0     0     0
           c0t6005076802810997280000000000002Bd0  ONLINE       0     0     0
           c0t6005076802810997280000000000002Cd0  ONLINE       0     0     0
           c0t6005076802810997280000000000002Dd0  ONLINE       0     0     0
           c0t6005076802810997280000000000002Ed0  ONLINE       0     0     0
           c0t6005076802810997280000000000002Fd0  ONLINE       0     0     0
           c0t60050768028109972800000000000030d0  ONLINE       0     0     0
           c0t60050768028109972800000000000031d0  ONLINE       0     0     0
           c0t60050768028109972800000000000032d0  ONLINE       0     0     0
           c0t60050768028109972800000000000033d0  ONLINE       0     0     0
           c0t60050768028109972800000000000034d0  ONLINE       0     0     0
           c0t60050768028109972800000000000035d0  ONLINE       0     0     0
           c0t60050768028109972800000000000036d0  ONLINE       0     0     0
           c0t60050768028109972800000000000037d0  ONLINE       0     0     0
           c0t60050768028109972800000000000038d0  ONLINE       0     0     0
           c0t60050768028109972800000000000039d0  ONLINE       0     0     0
           c0t6005076802810997280000000000003Ad0  ONLINE       0     0     0
           c0t6005076802810997280000000000003Bd0  ONLINE       0     0     0
           c0t6005076802810997280000000000003Cd0  ONLINE       0     0     0
           c0t6005076802810997280000000000003Dd0  ONLINE       0     0     0
           c0t6005076802810997280000000000003Ed0  ONLINE       0     0     0
           c0t6005076802810997280000000000003Fd0  ONLINE       0     0     0
           c0t60050768028109972800000000000040d0  ONLINE       0     0     0
           c0t60050768028109972800000000000041d0  ONLINE       0     0     0
           c0t60050768028109972800000000000042d0  ONLINE       0     0     0
           c0t60050768028109972800000000000043d0  ONLINE       0     0     0
    
    errors: No known data errors
    #root:~# zpool status SID1arch
      pool: SID1arch
     state: ONLINE
      scan: none requested
    config:
    
         NAME                                     STATE     READ WRITE CKSUM
         SID1arch                                 ONLINE       0     0     0
           c0t60050768028109972800000000000013d0  ONLINE       0     0     0
           c0t60050768028109972800000000000014d0  ONLINE       0     0     0
           c0t60050768028109972800000000000015d0  ONLINE       0     0     0
           c0t60050768028109972800000000000016d0  ONLINE       0     0     0
    
    errors: No known data errors
    #root:~# zpool status SID1redo
      pool: SID1redo
     state: ONLINE
      scan: none requested
    config:
    
         NAME                                     STATE     READ WRITE CKSUM
         SID1redo                                 ONLINE       0     0     0
           c0t60050768028109972800000000000012d0  ONLINE       0     0     0
    
    errors: No known data errors
  • 22. Re: simple processes are very slow at intervals
    800381 Explorer
    Currently Being Moderated
    Looking at the dtrace kernel hotspots, I'm wondering what driver is calling "drv_usecwait". I poked around a bit on the Solaris 11 boxes I have access to (no T-series, unfortunately...) and not one call was made to that. The man page says it's done by a driver that needs to wait for something, and it's a busy wait done spinning on the CPU. And from what you provided, other than cpu_halt, the calls to drv_usecwait are a huge fraction of where the kernel is spending its time.

    But if all the context switching and such is going on all the time, what is triggering the slowdowns? If it is ZFS ARC flushing, maybe chaning logbias to latency for all ZFS file systems would help?

    I still think you need to get the finer-grained sar data, then go through all the data collected during one of the slowdowns and see what's happening. Is there a bunch of IO to a certain LUN at those times, for example. Maybe you have an intermittent hardware problem on one of your paths? Is system CPU utilization pegging at 100% or close to it while user CPU utilzation drops to near zero?

    You're right about the context switching seeming to be extremely high, but is that a cause or a symptom?

    Another thought: how many machines is this happening on?
  • 23. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated
    drv_usecwait might it be the oracle BG vkmt processes which are running at highres_tick = 1ms doing multiple nanosleep sys calls, thou thou then they ought to be 100% cpu busy if usecwait is busy waiting and they are not. Are considering trying to turn off highres_tick, which seems to have changed default to on from previous Solaris versions thus making Oracle Virtual Time Keepers go from 10ms cycles to 1ms. Also considering section IV. Platform sleeps granularity of this link http://andreynikolaev.wordpress.com/2011/10/25/mutex-waits-part-ii-cursor-pin-s-in-oracle-11-2-mutexwait_scheme0-steps-out-of-shadow/

    True ctx sw f.ex. from ZFS daemons & oracle VKTM processes seems to going on even under relative light load which is also why I considering trying with former def. highres_tick = 0 ie. 100Hz timer resolution.

    ZFS BCP for Oracle is followed and that recommends different settings of latency versus throughput for various ZFS datasets, but yes it could be tested.

    I might have to sample more data during slow downs, problem is it might be hard to run utile because of slow downs :) Don't think it's a HW issue on a IO path, that I woiuld have expected to generate syslog events. Also SAN admin don't see issue in network or on array.

    No don't see cpu utilization pegging which indicates to me that time is wasted rather than spend processing, ie. kernel space lock contention maybe resumefrom_idle is often the highest called.

    Got another Oracle DBMSs on a set of Solaris 10 T5x40 boxes also on ZFS datasets, will compare ctx sw numbers with them. Numbers might not be an issue, Oracle Support doesn't seem scared of the numbers.

    This is happening on both T4-4 in this new data guard setup.
  • 24. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated
    Ok so far, we patched to latest SRU 7.5.0 on request of Oracle Solaris Tema, just to role out possible already fixed problems (some VM patches in SRU 4.x I believe).

    Then our DBAs ran their load again last night like they did last time issue were found. No issue were seen this time, but they are not totally convinced yet, as they feel issues first might happend when server has been run for a while like previously. So we are not totally in the clear yet. Also when it's not perfectly clear what was the issue it's harder to feel confident problem is solved.

    Next step, if issues still show up, we can think up is to try and revert hires_tick to 10ms as we read 'IV. Platform sleeps granularity' from this article: http://andreynikolaev.wordpress.com/2011/10/25/mutex-waits-part-ii-cursor-pin-s-in-oracle-11-2-mutexwait_scheme0-steps-out-of-shadow/

    Seems both did solaris 11.1 default now to 1000Hz timer resolution rather than earlier 100Hz (read in an article I can remember which now) and also did Oracle in 11.2.0.2.x change the hole mutex architecture so it's less cpu intensive aka from busy-waiting to other sort signaling mutexs. Might be that such a new mutex architecture possible is implemented via kernel features which possibly could cause symptoms like slowing CLIs when more work are thrown at Oracle aka more threads/core are applied to work, like Andrey seem to indicate. If this is the case, is it then a good idea to run Oracle with enableNUMA_support = true, ie. might this sooner pressure more threads/core to work thus cause effect like seen with Andrey's test where '1000Hz timer' response times seems to increase dramatical with more than loading 2 threads/core... Just wondering... thou I would assume/expect load testing as part of QA testing by both Oracle and Solaris team on latest their latest new default settings :)

    Will update when more findings are available...
  • 25. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated
    Not entirely true about hires_tick, it still default 10msec in Solaris 11.1 but read this blog https://blogs.oracle.com/sistare/entry/high_resolution_timeouts which says:

    The default resolution of application timers and timeouts is now 1 msec in Solaris 11.1, down from 10 msec in previous releases. This improves out-of-the-box performance of polling and event based applications, such as ticker applications, and even the Oracle rdbms log writer.

    ...

    Here are some exceptions for which the default resolution is still 10 msec.

    - The thread scheduler's time quantum is 10 msec by default, because preemption is driven by the clock thread (plus helper threads for scalability). See for example dispadmin, priocntl, fx_dptbl, rt_dptbl, and ts_dptbl. This may be changed using hires_tick.

    - The resolution of the clock_t data type, primarily used in DDI functions, is 10 msec. It may be changed using hires_tick. These functions are only used by developers writing kernel modules.

    - A few functions that pre-date POSIX CLOCK_REALTIME mention SCCLK_TCK, CLK_TCK, "system clock", or no clock domain. These functions are still driven by the clock thread, and their resolution is 10 msec. They include alarm, pcsample, times, clock, and setitimer for ITIMER_VIRTUAL and ITIMER_PROF. Their resolution may be changed using hires_tick.

    ...

    How does this help the Oracle log writer? Foreground processes post a redo record to the log writer, which releases them after the redo has committed. When a large number of foregrounds are waiting, the release step can slow down the log writer, so under heavy load, the foregrounds switch to a mode where they poll for completion. This scales better because every foreground can poll independently, but at the cost of waiting the minimum polling interval. That was 10 msec, but is now 1 msec in Solaris 11.1, so the foregrounds process transactions faster under load. Pretty cool.
  • 26. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated

    Seems we solved the issue by patching Solaris 11.1 from SRU 3.4.1 to latest SRU 7.5.0, somewhere inbetween maybe a VM patch helped (Support thought so) Anyway have not seen the issue since patching to 7.5.0.

  • 27. Re: simple processes are very slow at intervals
    986141 Newbie
    Currently Being Moderated

    Turned out that it most properly wasn't the sru7.5.0 that fixed the issue.

     

    Still experienced issues, specially when memory utilization got above 90%.

     

    Currently we assume this to have been due to a new feature with 2GB large memory pages, which has been turned off by:

     

    disable 2G pages. Add the following line to /etc/system and reboot:

    ---

    set disable_ism_large_pages=0x74

     

    Disabling 2G pages dramatically reduced our memory utilization (30-40%) at the same time and doesn't seem to have affect perf.

    The 2G pages is a recent addition, and in some cases it may cause performance issues, when it really should have improved performance a bit according to kernel perf. team.

     

     

    To check if you use 2G pages, do a pmap -xs f.ex. on ora_vktm_SID process(es):

     

    ps -fuoracle|awk '/ora_vktm_/{print $2}'| while read pid; do pmap -xs $pid | grep 2G; done

     

    as described here: "Oracle Database Startup And SGA Value Change Takes A Very Long Time To Complete", [Doc ID 1475621.1]

  • 28. Re: simple processes are very slow at intervals
    user12024958 Newbie
    Currently Being Moderated

    Hi all

    Had the same issue on T4-1 (Solaris10 + Oracle10.2.0.4). Lockstat shows a lot of "page_freelist_coalesce()" syscalls.

     

    The lack of available large pages will cause the system to attempt to create one when requested by coalescing smaller pages together. This often involves page relocation in order to acquire the linear physical address space necessary to create the large page. If the rate of large pages being requested is high, the system may experience performance issues due to this page coalescing operation.

    This is exhibited by high system time and possibly high lock contention if multiple threads are attempting to create a large page in the same memory node.

     

    Workaround is

    set mpss_coalesce_disable=1

    in /etc/system and reboot or

    'mpss_coalesce_disable/W 1' | mdb -kw

    without reboot

1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points