Infrastructure Software

Sure, by all means use hugepages here.  Especially on a small machine, using hugepages frees some resources and improves overall performance.

Tommy,

Many thanks for a quick response.

Is there a way I can prove/justify/quantify the advantage of hugepages on machines with 4-6 GB RAM?

I am more comfortable with Oracle Database but in general, I like to build small test cases that I can run and see the results myself.

In this case, what should I be looking for? For e.g. if I don't use hugepages on a linux server with 4 GB RAM (with Oracle Database 11gR2

using ASMM), will it show any kind of higher resource consumption? If enable hugepages, where can I see the "savings" / "improvements" ?

It would be great if I can get some pointers to put me in right direction.

Thanks in advance.

There are a few things you might want to consider about kernel Hugepages.

Simply put, memory on most OS's including Linux is managed in memory pages. The standard memory page size of the linux kernel on Intel x86 and x64 architecture is 4 KB. This also applies to POSIX shared memory using /dev/shm. Kernel Hugepages instead use 2 MB pages.  Each page in memory needs to be mapped in a page table, which requires additional memory.

For instance, on a server with 16 GB of RAM:

16 GB / 4 KB = 4194304 pages

16 GB / 2 MB = 8192 pages

You can see the size of the page table using: grep PageTable /proc/meminfo

It is quite obvious that kernel Hugepages uses a smaller page table and can take advantage of the TLB cache to improves performance when mapping virtual to physical memory. Check the following: Quantifying Hugepages Memory Savings with Oracle Database 11g

Kernel Hugepages, unlike standard memory and POSIX shared memory cannot be swapped to disk and is allocated and reserved at system startup. But in some cases, like Oracle database this can actually be desired. If I remember correctly, Red Hat recommends a system with at least 16 GB of RAM for enabling kernel Hugepages.  The default POSIX shared memory (/dev/shm) is 50 % of physical RAM and allocated on demand. Oracle database 11g requires /dev/shm to use Automatic Shared Memory (AMM). If your Oracle database(s) uses a SGA of 6 - 8 GB, kernel Hugepages can be a huge performance advantage.

Dude,

Many thanks for your response.

While I am grateful for the explanations you have provided, my original question was about justifying hugepage usage on machines with 4-6 GB RAM.

I think, to some extent, I have understood how hugepages can help when the server RAM is more than 8 GB. The link you have provided and few others

that I have read, also explain the symptoms and issues faced when not using hugepages on servers with more than 8 GB RAM.

I guess what I am trying to understand is whether the large size of pagetables causes same/similar issues when RAM is small (say 4-6 GB) as those

caused when not using hugepages on servers with more than 8 GB RAM). Or is there any other advantage that can be gained by using hugepages?

It might help if I share a story (I say story because I only heard about it and that too very limited information so...) that prompted me to ask this question.

One of my colleagues mentioned recently that when he tried to set up a linux server with 4 GB RAM (I know.Iknow..Iknow...sounds weird that anybody can say a machine with 4 GB RAM can be called a "server" in 2013...), he configured hugepages on it. Now, I am not aware of the exact configuration he used but he said that the machine was extremely unstable after starting it with hugepages. When he disabled hugepages, the machine started working normally.

Now, I understand, there are many "holes" in this story but this is where I would like to understand what all could have gone wrong in this story.

Hence, I asked earlier if there was a way I can build some kind of test case to prove the benefit of hugepages on servers, having either 4GB RAM or 48GB RAM.

Hope I have managed to make my point more clear.

Thanks in advance.

Hugepages is shared memory. You cannot convert all phsycial RAM from standard or conventional to Hugepages. There are only a few applications, such as Oracle for instance that actually use shared memory. Kernel Hugepages, like I already explained are reserved during system startup and hence cannot be used for anything that relies on conventional memory. If you have a system with only 4 GB of RAM, how much memory can you allocate for exclusive use of kernel Hugepages? There will most likely not be enough conventional memory left for the system to perform reasonably. Which should explain the bad experience of your colleague.

I think the recommendation of Red Hat, which is in the documentation, if I remember correctly, not to configure kernel Hugepages unless you have at least 16 GB of RAM makes sense. Actually in the past, the Hugemem kernel was required to use all the memory in system configurations with more than 16GB of memory.

On the other hand the memory page size also defines the smallest unit of data, which could be inefficient depending on memory fragmentation and the software you use.

Some more info: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt

Users who wish to use hugetlb page via shared memory segment should be a member of a supplementary group and system admin needs to

configure that gid into /proc/sys/vm/hugetlb_shm_group.

Users can use the huge page support in Linux kernel by either using the mmap system call or standard SYSV shared memory system calls (shmget, shmat).

Or how about a video with someone from the Oracle Linux kernel engineering team, who also talks about Transparent Hugepages introduced in OL 6.

GregMarsden Hugepages = Huge Performance on Linux - YouTube

Dude,

Thanks once again for your help.

This may sound crazy but I am (once again) getting confused with your statement "Hugepages is shared memory". What exactly does that mean?

I am confused because I thought /dev/shm is shared memory. My understanding is /dev/shm is shared memory, which can be shared between multiple processes,

which allows Oracle to implement AMM to manage both PGA and SGA as single entity. But it is also true that Hugepages are not compatible with AMM.

Also, if we go by Red Hat's advice of using hugepages provided RAM is 16GB or more, it does not exactly go with what Tommy mentioned above.

I know I am deviating from my original question here so apologies for that but I would really like to understand this beast...

Let me try to explain what I understood so far from your posts.

Memory page size is the smallest unit in which memory can be allocated to the processes that require the memory. In latest linux kernels i.e. 2.6.x (or even 3.x),

the default memory page size is 4KB. This means, all processes would acquire necessary memory in chuks of 4KB. The OS keeps track of memory page allocation

in lists called page tables (in memory). This means the page tables themselves will occupy space in RAM. More the entries in this list, more the size of the page tables.

As the number of items on page tables grow, CPU (??) has to do more work to be able to reach the required memory page. This, in effect, can cause performance issues

when running applications like Oracle Database, configured with large SGA, on machines with large amount of RAM.

Hugepages is a way to be able to define a bespoke memory page size (bigger than the default 4KB). The bigger size of memory pages means less entries in page tables

and easier (for CPU) to reach the required memory page thereby resulting in  lesser resource utilization and better performance. However, because Hugepages are memory

pages with non-default size, only some processes/applications can use them and that too when explicitly told to do so (USE_LARGE_PAGES in 11.2.0.2 onwards).

But then I look at the blog post by Kevin Closson that you have pointed to and he shows (the test case/working example that I always find easy to understand) how increasing

DB connections (in dedicated server mode) increases page table size. That example tells the benefit of Hugepages on PGA memory usage (the DB connections taht just connect and do nothing).

So does it mean Oracle, when configured, uses hugepages for both SGA and PGA ? If Hugepages is "shared memory", why can it not be "shared" between SGA and PGA, like AMM using /dev/shm?

If I am not mistaken, a dedicated server DB connection that does nothing would consume very little PGA memory. So does it mean Kevin's example only points out the significant load on OS (page table size specifically)

when having to handle large number of DB processes with default memory page size as compared to much reduced load on OS (page table size) when using Hugepages?

Once again, am happy to be corrected and apprciate your help.

Thanks in advance

Shared memory and semaphores, including kernel Hugepages in Linux belong to System V IPC, which can be monitored using the ipcs command. /dev/shm or ramfs is POSIX shared memory, which maps shared memory to files using a virtual shared memory filesystem in RAM. The two are different and as far as I know Oracle uses shared memory only for SGA and not PGA. The PGA contains data and control information for every server process and is nonshared memory. An Oracle instance can use either Hugepages or /dev/shm for SGA, but not both at the same time. You can however use different shared memory concepts for different Oracle instances on the same server, like AMM for ASM and ASMM with Hugepages or conventional 4k memory pages for ORCL. If you want to configure SGA for Hugepages, you need to unset the MEMORY_TARGET and MEMORY_TARGET_MAX parameter for AMM - setting to 0 i s not good enough.

Dude! wrote:

Shared memory and semaphores, including kernel Hugepages in Linux belong to System V IPC, which can be monitored using the ipcs command. /dev/shm or ramfs is POSIX shared memory, which maps shared memory to files using a virtual shared memory filesystem in RAM. The two are different and as far as I know Oracle uses shared memory only for SGA and not PGA. The PGA contains data and control information for every server process and is nonshared memory. 

Dude,

If Oracle uses shared memory only for SGA then how is it possible that the example mentioned in the Kevin's blog post (which you have referred to earlier) benefits from hugepages? Did I understand his test case wrong completely? As far as I can understand,

he demonstrated that creating large number of db connections (that do nothing) without using hugepages caused huge page table size and the page table size dropped significantly when the test was rerun with hugepages. To me, that test case did not involve

any SGA usage (other than the minimal default one).

Sorry I cannot see where the link mentions anything about PGA? From what I understand, the example simply shows the impact 500 connections have in regard to the resulting huge size of the memory page table, showing which difference Hugepages can make.

The UGA cannot be stored in the PGA when using a shared server connection because the PGA is specific to a single process. Therefore, the UGA is stored in the SGA when using shared server connections, enabling any shared server process access to it. When using a dedicated server connection, the UGA is stored in the PGA.

Dude,

The case in link uses following script (mentioned in another linked blog there)

====================================================================================================================================================================================================

Using Memory and Then Allocating HugePages (Or Die Trying)
I purposefully booted my system with no hugepages allocated in /etc/sysctl.conf (vm.nr_hugepages = 0). I then booted an Oracle Database 11g instance with sga_target set to 8000M. Next, I fired off 500 dedicated connections using the following goofy stuff:

$ cat doit  cnt=0  until [ $cnt -eq 500 ]  do  sqlplus rw/rw @foo.sql &  (( cnt = $cnt + 1 ))  done    
$ cat foo.sql  HOST sleep 120  exit;  

====================================================================================================================================================================================================

If one is connecting using sqlplus and doing nothing, would it consume any SGA? It is mentioned that the connections are in dedicated server mode which means even the UGA will be in PGA.

I don't see how that script can use SGA (I can see that consuming PGA though). Hence my doubt about hugepages and PGA.

Perhaps your assumption is that connections using dedicated server mode do not use any SGA or that PGA is allocated using shared memory. Again, as far as I know, PGA is not allocated using SGA or shared memory. You will have to ask the blog owner whether or not the server was running in shared or dedicated server mode, or perhaps test it yourself. How much memory user sessions will use out of SGA or PGA is also irrelevant to the question of understanding or questioning the use of Hugepages. The performance and memory usage benefit of Hugepages is simply depending on the total size of your SGA, which you or the DBA set by Oracle instance parameters.

If you server does not have much RAM or the database does not have a large memory footprint then the use of Hugepages or AMM (/dev/shm) is a question of convenience. I would consider a SGA of 4 GB to be a valid candidate for Hugepages. The OS will however need at least 1 GB per CPU to function. Keep in mind that /dev/shm is 50 % of physical RAM, can be swapped and is allocated on demand, unlike kernel Hugepages, which cannot be swapped and is contiguous memory reserved for shared memory purpose during system restart. Oracle Linux 6 introduces a new concept of transparent Hugepages, but Oracle cannot use this this type of memory for SGA.

Dude! wrote:

Perhaps your assumption is that connections using dedicated server mode do not use any SGA or that PGA is allocated using shared memory. Again, as far as I know, PGA is not allocated using SGA or shared memory. You will have to ask the blog owner whether or not the server was running in shared or dedicated server mode, or perhaps test it yourself. How much memory user sessions will use out of SGA or PGA is also irrelevant to the question of understanding or questioning the use of Hugepages. The performance and memory usage benefit of Hugepages is simply depending on the total size of your SGA, which you or the DBA set by Oracle instance parameters.

Dude,

Thanks for your patience. I am definitely not assuming that connections in dedicated server mode do not use any SGA. I was only talking about the specific test case used by Kevin where he clearly states that a db connection is created using sqlplus and the sqlplus session does nothing that needs access to SGA.

Anyways, thanks once again for your help on this. I can't say my original question is answered but I guess I have managed to "move forward" in understanding Hugepages.

As for my original question, if I consider a machine with 4GB RAM used for running Oracle Database, typically about 1.6GB would be the SGA size. A 1.6GB SGA with default 4KB page size would mean 419430 memory pages whereas with 2MB hugepage size, the number of memory pages would be about 820. I guess even though the difference in number of pages in this case is quite big, the page table size probably remains manageable (by OS), which might mean any performance benefits are not noticable.

Does that make any sense or is it just a load of misguided nonsense?

I don't know if there was any performance increase between ASMM with kernel Hugepages and AMM with a SGA of just a couple of Gigs. But I would question that running any modern OS with just 2 GB of RAM is sooner or later going to be an issue. Hence I would not use kernel Hugepages in your described scenario for the sake of OS availability, at least not without installing more physical RAM. There is probably some threshold, which depends on usage, that defines whether a smaller size of the page table becomes the lesser evil than more efficient use by smaller memory pages.

Message was edited by: Dude! The OS does not use shared memory Hugepages, only some applications like Oracle. So there I think you have another criteria to make the decision or answer your question. Because if your system has 4 GB of RAM and you setup 2 GB for kernel Hugepages, it will leave your OS limited to 2 GB of RAM. So you can probably open Firefox and that will be the end of the rope.

Dude,

Many thanks for your patience and helpful answers.

I am going to close this thread now as I feel I am now really less confused about the OS memory usage...thanks to your (& Kevin's) helpful responses.