Infrastructure Software

I would like to enable large heap pages for an application that may be experiencing high TLB miss rates, but I am having difficulty getting the tools to manipulate the page size to work correctly.

On SPARC, I can get the stack to use large pages, but not the heap. On x86, I cannot get either the stack or the heap to use large pages.

Here is a transcript of some experiments to adjust the page size for 'cat', first run on SPARC, and then repeated on an x86 machine.


user@sparc ~> cat /etc/release
Solaris 10 1/06 s10s_u1wos_19a SPARC
Copyright 2005 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 07 December 2005
user@sparc ~> cat
^Z
Suspended
user@sparc ~> pmap -s `pgrep cat` | egrep "stack|heap"
0002A000 16K 8K rwx-- [ heap ]
FFBFE000 8K 8K rw--- [ stack ]
user@sparc ~> fg
cat
^C
user@sparc ~> pagesize -a
8192
65536
524288
4194304
user@sparc ~> ( setenv LD_PRELOAD mpss.so.1 ; setenv MPSSHEAP 4M ; setenv MPSSSTACK 4M ; cat )
^Z
Suspended
user@sparc ~> pldd `pgrep cat`
23276: cat
/usr/lib/mpss.so.1
/lib/libc.so.1
/lib/libgen.so.1
/platform/sun4u/lib/libc_psr.so.1
user@sparc ~> pmap -s `pgrep cat` | egrep "stack|heap"
0002A000 16K 8K rwx-- [ heap ]
0002E000 3912K - rwx-- [ heap ]
FF800000 4096K 4M rw--- [ stack ]
user@sparc ~> fg
( setenv LD_PRELOAD mpss.so.1; setenv MPSSHEAP 4M; setenv MPSSSTACK 4M; cat )
^C
user@sparc ~> ppgsz -o heap=4M,stack=4M cat
^Z
Suspended
user@sparc ~> pmap -s `pgrep cat` | egrep "stack|heap"
0002A000 16K 8K rwx-- [ heap ]
0002E000 3912K - rwx-- [ heap ]
FF800000 4096K 4M rw--- [ stack ]
user@sparc ~> fg
ppgsz -o heap=4M,stack=4M cat
^C
user@sparc ~>

Here, using either mpss.so.1 or ppgsz, I can adjust the stack page size, but not the heap.


The same set of tests run on x86:

user@x86 ~> cat /etc/release
Solaris 10 6/06 s10x_u2wos_09a X86
Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 09 June 2006
user@x86 ~> cat
^Z
Suspended
user@x86 ~> pmap -s `pgrep cat` | egrep "stack|heap"
08047000 4K 4K rw--- [ stack ]
0806A000 12K 4K rw--- [ heap ]
user@x86 ~> fg
cat
^C
user@x86 ~> pagesize -a
4096
2097152
user@x86 ~> ( setenv LD_PRELOAD mpss.so.1 ; setenv MPSSHEAP 2M ; setenv MPSSSTACK 2M ; cat )
^Z
Suspended
user@x86 ~> pldd `pgrep cat`
19155: cat
/usr/lib/mpss.so.1
/lib/libc.so.1
/lib/libgen.so.1
user@x86 ~> pmap -s `pgrep cat` | egrep "stack|heap"
08047000 4K 4K rw--- [ stack ]
0806A000 12K 4K rw--- [ heap ]
0806D000 1612K - rw--- [ heap ]
user@x86 ~> fg
( setenv LD_PRELOAD mpss.so.1; setenv MPSSHEAP 2M; setenv MPSSSTACK 2M; cat )
^C
user@x86 ~> ppgsz -o heap=2M,stack=2M cat
^Z
Suspended
user@x86 ~> pmap -s `pgrep cat` | egrep "stack|heap"
08047000 4K 4K rw--- [ stack ]
0806A000 12K 4K rw--- [ heap ]
0806D000 1612K - rw--- [ heap ]
user@x86 ~> fg
ppgsz -o heap=2M,stack=2M cat
^C
user@x86 ~>

In this case, I cannot adjust either the heap or the stack size with either method.

The sparc machine is an Ultra IIe. The x86 machine is an AMD Opteron(tm) Processor 146.

Any insight would be appreciated.
Thanks.