This discussion is archived
1 2 Previous Next 24 Replies Latest reply: Sep 25, 2012 5:10 AM by rukbat Go to original post RSS
  • 15. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    801904 Newbie
    Currently Being Moderated
    that was it.

    Give his man a medal, the ony way to get HugePages to work is to ENTIRELY DELETE/REMOVE memory_target and memory_max_target from pfile and then create spfile from it, because you cant delete entries from SPFILE directly....

    Setting them to zero does not work even though the manual says is does.

    THANKS DUDE !!!!!!

    Working now, using hugepages !!

    the show parameter mem shows both of the above set to 0 as if I did not remove them at all

    so what is the difference here?

    tell me, flame or no flame, is this documented or a bug?
    I dont recall anywhere in the doc stating that removing the memory parameter from pfile is different to setting it to =0 even though after startup they show same values !!

    I will updatge my Oracle Database Forum post with credit to you ! Thanks!!!!
  • 16. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    user5749629 Newbie
    Currently Being Moderated
    Just wanted to provide another perspective - hope it makes sense to the readers.

    AMM and its dynamic management of the memory provide great peace of mind. Since we started using it a few years ago we haven't had any more "out of memory" issues. Turning it off and self-managing SGA and PGA means taking on this responsibility - so would probably work well on a DB where the load is pretty constant - but will incur extra headaches if it varies.

    Yes, AMM precludes the use of Huge Pages - maybe one of the Oracle insiders can explain why making these 2 hugely beneficial technologies work together has not been a priority for Oracle? :) - so you have to accept the loss of Huge Pages if you go with AMM - but with only 64GB - it's not as big a loss as if you had 1TB for example.

    Yes, Linux takes 50% of RAM for tmpfs which seems a lot. We were initially tempted to cut it down to 10-20% - but then we saw that OEL makes extremely good use of RAM for caching data and speeding up IO. On large servers we have seen IO double from what the disk arrays can physically do (and this is on sequential IO where disks do pretty well) - when we gave Linux the full 50%. Results will of course vary depending on how much RAM there is in total - but the take home message for us was not to try to take away from Linux in the hope that Oracle will make better use of it. The 50-50% split was a good one.

    In summary - AMM and the 50-50% default is quite good - and much easier - unless you plan on having an exceptionally large SGA where going with Huge Pages and manual memory configuration might be justifiable - but will require vigilance and monitoring.

    A humble opinion. Good luck.
  • 17. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    Dude! Guru
    Currently Being Moderated
    There is probably a good reason why AMM is incompatible with Kernel Hugepages:

    AMM or /dev/shm can be swapped to disk, physical RAM allocated as needed.
    Hugepages will only remain in physical RAM can cannot be swapped to disk. Physical RAM reserved at system startup.

    AMM relies on the default 4k page size. Hugepages can be 2 up to 256 MB per page. If you have an instance that uses more than 8 GB you should notice quite a performance increase and save memory since the kernel will also need less memory to manage memory pages.

    For instance:

    Standard memory pages:

    17 GB Oracle SGA:
    4400000 pages (4k)
    TLB can only hold 1024 entries, huge performance penalty.

    Hugepages:

    17 GB Oracle SGA:
    8700 pages (2m)
    huge improvement over AMM.
  • 18. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    user5749629 Newbie
    Currently Being Moderated
    Hi Avi,

    Thanks for the tip. The "remount" worked. However - I seem to be running into an old Oracle bug where MEMORY_TARGET cannot exceed 256GB. It was broken in OEL 5.5 and fixed in 5.7. Now with 6.3 it seems broken again.

    I have 1 TB of RAM and wanted TMPFS set at 855GB so that I can set MEMORY_TARGET to 850GB. Simple AMM, no hugepages for starters.

    I issued the "remount" (only with just a "G" - it didn't take "GB" in the syntax). TMPFS was resized - but starting Oracle with 250GB worked but with 260GB didn't.

    Any ideas? This is OEL 6.3 and ASM/Oracle 11.2.0.3 .

    Below is the "df -h" before and after the remount. The first giving TMPFS 1/2 of the RAM (497G). The second giving TMPFS the full 855GB.

    Following is the starting of Oracle with 250GB of MEMORY_TARGET (worked) followed by 260GB (failed). Thanks !


    df -h
    tmpfs 497G 887M 496G 1% /dev/shm

    mount -o remount,size=855G /dev/shm

    df -h
    tmpfs 855G 887M 855G 1% /dev/shm


    sqlplus / as sysdba

    SQL> startup pfile=/home/oracle/init.ora

    ORACLE instance started.

    Total System Global Area 2.6724E+11 bytes
    Fixed Size 2244992 bytes
    Variable Size 1.2187E+11 bytes
    Database Buffers 1.4496E+11 bytes
    Redo Buffers 409980928 bytes
    Database mounted.
    Database opened.

    SQL> shutdown immediate
    Database closed.
    Database dismounted.
    ORACLE instance shut down.


    SQL> startup pfile=/home/oracle/init.ora

    ORA-27123: unable to attach to shared memory segment
    Linux-x86_64 Error: 22: Invalid argument
    Additional information: 7
    Additional information: 589829
    Additional information: -536870912
  • 19. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    Dude! Guru
    Currently Being Moderated
    Why do you use AMM with that amount of memory? You are trashing a huge part of performance and memory. The size of the memory page table alone must be incredible. With AMM you will have a standard page frame for every 4k of RAM. Sorry, but your configuration is not reasonable.
  • 20. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    Avi Miller Guru
    Currently Being Moderated
    user5749629 wrote:
    Thanks for the tip. The "remount" worked. However - I seem to be running into an old Oracle bug where MEMORY_TARGET cannot exceed 256GB. It was broken in OEL 5.5 and fixed in 5.7. Now with 6.3 it seems broken again.
    Was this a DB bug or a Linux bug?
    I have 1 TB of RAM and wanted TMPFS set at 855GB so that I can set MEMORY_TARGET to 850GB. Simple AMM, no hugepages for starters.
    Yeesh. We recommend hugepages for anything over 8GB, so it's definitely required for 1TB. Either way, I'd open an SR with Oracle Support so they can troubleshoot whether the issue is in DB or the kernel. You don't say if you're running the RHCK (2.6.32) or UEK2 (2.6.39) kernel on OL6U3. Might want to try switching to the other one and seeing if you still have the issue.
  • 21. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    user5749629 Newbie
    Currently Being Moderated
    Avi - thank you so much for pursuing this. We are on UEK2. The Oracle OEL folks really believe it's a better way to go - so we are giving it a try.

    Linux (server name) 2.6.39-200.29.3.el6uek.x86_64 #1 SMP Tue Aug 28 13:03:31 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

    Avi and Dude - your points are well taken. I do intend to use hugepages eventually. It's just that it's additional tinkering - and at this point I am not sure how best to split the 1 TB between Linux / SGA / PGA - so I went with simple AMM just to get my toe in the water - and ran across the old familiar bug.

    Not sure if the 256G limitation is Linux or Oracle - we opened an SR at the time with 5.5 & 11.2.0.2 and never got a resolution until we happened to try 5.7 and the problem went away.

    As an aside - forgoing AMM (no hugepages still) and setting SGA=100G and PGA=750G works. So it seems like the sensitivity to the 256G is SGA driven. Keep it under 256G and it seems OK. Then again - I haven't yet run jobs that truly required the full PGA - so the sum of both might still be an issue. We'll see.

    This is a test server. We are planning to order servers with 2TB - so taking advantage of it all and dividing it correctly among the 3 players is the goal. Once we establish an optimal SGA size - we'll tackle the hugepages setup.

    Thanks again !
  • 22. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    user5749629 Newbie
    Currently Being Moderated
    Avi and Dude -

    I opened an SR about the inability to allocate memory_target > 256GB - but we haven't made much headway so far - so I decided to try huge pages in the meantime.

    With 1TB of RAM I allocated 450GB to huge pages (225,000 pages of 2M), and allocated an SGA of 400GB.

    This worked fine - but here's what happened. Our tables are multi-TB, so while Oracle uses the SGA as best it can - most IO is from disk, and having lots of RAM allows OEL / UEK to assist the reads in a meaningful way. The OS is really capable that way. You can get read rates of as much as 2X the physical IO capacity of the IO subsystem of the server through the OS caching.

    The thing is - this requires leaving the OS ample free RAM - so typically the default tmpfs setting of 50% is good.

    However - here - I took away almost half the RAM for huge pages - so there is much less RAM for the OS to use as assist - and read rates are about 1/2 as before.

    So the SGA might be more efficient - but the reads have taken a hit. Also - large queries seem to run slower when explicitly specifying SGA and PGA versus AMM - though more testing is required to confirm this.

    So this is the dilemma - I completely understand that AMM => 1k pages => inefficient memory management for the SGA. However - leaving the entire RAM for the OS to use as cache and for AMM to use /dev/shm as it pleases - seems to - runtime wise - seem to be extremely efficient.

    Thoughts? Suggestions? I am still pursuing being able to create a memory_target > 256GB - because it should work - efficient or not - but I also want to make sure I give huge_pages the proper testing.

    Also - maybe on future servers with 2TB of RAM - I can dedicate 1 TB to enhancing the OS cache effect and another 1TB to SGA/PGA where I can use huge_pages for the SGA part. Nevertheless - this is still subject to confirming that by manually specifying SGA and PGA sizes - we can get just as good performance as giving it all to AMM to split between the 2 as it sees fit.

    ??????????????????

    Thanks !
  • 23. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    Dude! Guru
    Currently Being Moderated
    I have no idea what happened to the OP starting this thread, but I would not recommend to hijack its content to address your very own problem. This is considered bad practice. I recommend to start a new thread and provide a link for reference if necessary.
  • 24. Re: /dev/shm on Oracle Linux 6.x to run Oracle 11g R2 - manual configuration?
    rukbat Guru Moderator
    Currently Being Moderated
    Moderator Action:
    This thread was begun in May 2012 and seems to have been abandoned by the O.P. within a three day span.

    It is now locked to prevent additional resurrection hijacks.

    @ O.P.,
    If you ever want it unlocked, you can post a separate new thread asking a moderator to have that done. (include a link to this thread in your unlocking request, so as to be precise on what you might want unlocked).
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points