I was curious as to how other customers of Exadata had implemented (or not) HugePages. As far as I understand, HugePages is the 'best practice' for Exadata, but this means you lose the benefits of Automatic Memory Management.
For us, we do not notice any swapping of SGA and our system is, like most, a little on the schizophrenic side: we have very heavy batch processing overnight and a combination of OLTP and BI reporting during the day. For me, the ability to use Automatic Memory Management to allow the database to tune its memory structures for this change in the processing outweighs the potential upside of HugePages. I suspect with Oracle pushing the later Exadata machines as DSS AND OLTP, surely AMM is the way to go?
In my opinion, the main benefit of automatic memory management is that it reduces initial configuration effort. You can use ASMM (automatic shared memory management) on Exadata, so the only major difference is that the split between PGA/SGA memory is not automatic. I've personally experiencesd extremely variable performance under AMM when the database suddenly decided to dump a large part of the buffer cache to do a sort. And in earlier Oracle versions, actually cause the system to hang due to aging out of the library cache.
I've seen much more consistent performance establishing static PGA and SGA targets, leaving a margin of free memory of course.
But results always vary by workload; this is a pretty simple case to test: run your workload with AMM, then configure hugepages and set PGA/SGA targets, using the memory you saved from using hugepages too. I think the forum members would be very interested in seeing your results.
I have seen variable performance too, especially inside the SGA when the shared pool and buffer cache attempt to kick the crap out of each other – though only on previous versions of the RDBMS. I find that 220.127.116.11 seems to handle its resizing better. Of course, a lot of that could be the sheer size of the memory footprint Exadata allows which reduces any such competition.
We originally had HugePages enabled when we first went live (on 11gR1), but when we switched to 11gR2, we found that the change in processing really benefitted from using AMM instead – not so much the batch processing, because we had configured the structures to benefit that – but moreso the BI/OLTP stuff. To be honest, I didn’t notice any benefit to HugePages when we used it – I still don’t see any swapping, etc, etc, even under heavy load and we’re pushing the limits of our V2 machine in terms of CPU, disk allocated, etc.
I know that Oracle are pushing the 'Exadata as OLTP' heavily now, so I'm curious as to why HugePages is still considered a 'best practice', I guess.
Two quick notes:
1. I would expect that, under Exadata with hugepages, you'd benefit from larger SGA/PGA sizes if nothing else. /proc/meminfo will show how much memory you're using for page tables, and you should see if much smaller with hugepages enabled.
2. The official word from Oracle is in MOS doc 1274318.1 (https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?type=DOCUMENT&id=1274318.1#verifyopersystemhugepages ). And yes the recommendation is to use hugepages for "more efficient use of memory and reduced paging".
I belive that hugepages are actually calculated and a default value is provided by onecommand. I understand it is highly recommended to use hugepages and disable AMM from 18.104.22.168 or so onwards. see Oracle Sun Database Machine Setup/Configuration Best Practices [ID 1274318.1] "Verify operating system hugepages count satisfies total SGA requirements"
You guys are sowing the seeds of doubt in my mind, especially given the fact that we're currently at 22.214.171.124 and plan to upgrade to 126.96.36.199 (maybe even 188.8.131.52, if it's out in time) in the New Year.
I've read the MOS documents, but they didn't convince me to give up AMM. Now that I have two gurus suggesting that HugePages are a real performance boon, I'm doubting myself and will have to review my earlier findings. Thanks, guys!
Slightly Offtopic is an issue we came across with AMM on a non exadata system but it also illustrates issues one can face with AMM. We found in one site there was a lot of pressure on the buffer cache , it was getting squeezed. We tracked it down to a parsing bug in the XML parser that was allocating memory but was not freeing the same. Due to this PGA allocated to various sessions was growing and Oracle automatically started to reduce SGA size to a point when the database performance as a whole was unacceptable. So the caveat with AMM wherever you use it is to always define minimum sizes and also investigate when Oracle starts hitting those minimums and not take it on faith that Oracle will not try to shoot itself in the foot.