7 Replies Latest reply on Feb 22, 2011 7:49 AM by 799157

    Oracle 10.2 ASM on AIX 5.3 compatibility with IBM DS4300 & EXP700 storage

    799157
      Hello

      We're in the process of expanding our storage (using IBM DS4300 and a number of EXP700s i.e adding a couple of new EXP700s) and as part of this want to look at use of ASM in preference to simply adding new or expanding existing mount points on our AIX 5.3 pSeries (we're currently totally file system based for our data files and that has it's own administration problems).

      I have been trying to find some form of Oracle compatibility matrix for ASM with the above storage to verify this is a viable option but with no luck yet after much searching on the Oracle pages and via Google.

      Can anyone here point me in the right direction? Are there any best practice guides for this configuration available?

      Many thanks

      Pete
        • 1. Re: Oracle 10.2 ASM on AIX 5.3 compatibility with IBM DS4300 & EXP700 storage
          Levi Pereira
          Hi,
          I have been trying to find some form of Oracle compatibility matrix for ASM with the above storage to verify this is a viable option but with no luck yet after much searching on the Oracle pages and via Google.
          Check this..
          Re: RAC Storage arrays  certified by Oracle

          Can anyone here point me in the right direction? Are there any best practice guides for this configuration available?
          Best Practices for Running an Oracle Database on an IBM Mid-Range Storage Subsystem
          http://levipereira.wordpress.com/2010/12/21/best-practices-for-running-an-oracle-database-on-an-ibm-mid-range-storage-subsystem/

          Best Practices for configuring IBM TotalStorag  DS4100/DS4300 Storage Systems: ORACLE 10g and ASM

          http://levipereira.wordpress.com/2011/02/16/best-practices-for-configuring-ds4100-ds4300-storage-systems-oracle-10g-and-asm/

          Regards,
          Levi Pereira
          1 person found this helpful
          • 2. Re: Oracle 10.2 ASM on AIX 5.3 compatibility with IBM DS4300 & EXP700 storage
            799157
            Hi Levi

            Many thanks for these - it should make very interesting reading and help significantly.

            Cheers

            Pete
            • 3. Re: Oracle 10.2 ASM on AIX 5.3 compatibility with IBM DS4300 & EXP700 storage
              Levi Pereira
              Hi Pete,

              If the thread was answered check the correct answer and close the thread.

              Regards,
              Levi Pereira
              • 4. Re: Oracle 10.2 ASM on AIX 5.3 compatibility with IBM DS4300 & EXP700 storage
                799157
                Hi Levi

                Well, I was reading through the documentation that you pointed me to and didn't want to close the thread if there I had follow on questions on this topic.

                For example, your best practices paper was very useful and definitely appears to answer the initial question concerning compatibility but it raises new ones such as what would be the best way to configure the two new EXP700s? As two failure groups so that ASM effectively provides the mirroring, or, to use IBM Storage Manager to do the mirroring and mirror each disk in one EXP700 with one in the other EXP700 and specify external redundancy to ASM. Are there any major pros and cons with the options such as if ASM does the mirroring then that results in double the write I/Os to the SAN compared to if the storage does the mirroring. (I hope that that makes sense).

                I also keep coming across stripe size references but if you just mirror the disks and pass themn to ASM which is doing its own form of striping then where are we needing to specify a stripe size? (Apologies if that is too basic a question but not having used Storage Manager nor ASM before then it is important to my understanding).

                Our hot spare disks are on the development EXP700 in order to switch in for any disk on the production EXP700s: I assume that the automatic recognition of failing main disks and substituting in a hot spare is done by the storage and is effectively invisible to ASM.

                With 14 new disks in each new production EXP700, is there an optimal way in which to present them ASM i.e. just provide 14 new (mirrored pairs of) disks and let ASM stripe over all of them or define groups of sizes 4 and 8 and is this more optimal in any way (assume for the moment that we don't need to have a separate group for anything such as an FRA, etc.) i.e. is a group of 14 disks better or worse than groups of (8 and 4 and 2) disks or (8 and 6) disks?

                Cheers

                Pete
                • 5. Re: Oracle 10.2 ASM on AIX 5.3 compatibility with IBM DS4300 & EXP700 storage
                  Levi Pereira
                  Hi Pete,

                  Very good ... you did great questions.
                  What would be the best way to configure the two new EXP700s? As two failure groups so that ASM effectively provides the mirroring, or, to use IBM Storage Manager to do the mirroring and mirror each disk in one EXP700 with one in the other EXP700 and specify external redundancy to ASM.
                  Are there any major pros and cons with the options such as if ASM does the mirroring then that results in double the write I/Os to the SAN compared to if the storage does the mirroring. (I hope that that makes sense).
                  Our hot spare disks are on the development EXP700 in order to switch in for any disk on the production EXP700s: I assume that the automatic recognition of failing main disks and substituting in a hot spare is done by the storage and is effectively invisible to ASM.
                  The hardware (storage) makes all works of redundancy, stripe and failure control without consuming CPU Host and in a way much faster instead of using the CPU Host with ASM.

                  With ASM Redundancy you can not use feature "Hot Spare", this is realy bad.

                  I recomend this you read this...
                  http://www.ardentperf.com/2010/07/15/asm-mirroring-no-hot-spare-disk/

                  Be careful to not switch a pig in a poke. Exchanging feature of Storage by feature of the ASM.

                  I also keep coming across stripe size references but if you just mirror the disks and pass themn to ASM which is doing its own form of striping then where are we needing to specify a stripe size? (Apologies if that is too basic a question but not having used Storage Manager nor ASM before then it is important to my understanding).
                  The more spread out is the data in Storage its probable performance gain is greater. (OLTP)

                  Four parameters are of great importance:

                  Allocation Unit (ASM)
                  Block Size (Database)
                  Segment Size (Storage)
                  Size of LUNS

                  ASM Configuration Allocation Units

                  Every ASM disk (LUN) is divided into allocation units (AU). An AU is the fundamental unit of allocation within a disk group. A file extent consists of one or more AU. An ASM file consists of one or more file extents. A database created under the constructs of ASM will be striped by default; as specified in the SAME methodology; i.e., the I/O load is evenly distributed and balanced across all disks within the diskgroup. Example with four disk, the database will be striped across four disks, and redundancy will be handled by the storage array (ASM external Redundancy).
                  The striping is done on a file-by-file basis, using a 1MB stripe size, as opposed to other LVMs that do striping and mirroring at a disk-volume level. ASM 1MB stripe depth has proved to be the best stripe depth for Oracle databases. This optimal stripe depth coupled with even distribution of extents in the diskgroup, prevents the occurrence of hot spots.

                  ASM always creates one-allocation-unit (AU) extents across all of the disks in a disk group. For a diskgroup with similarly sized disks, there should be an equal number of AU extents on every disk. A database file is broken up into file extents. There are two types of file extent distributions: coarse and fine.

                  For coarse distribution, each coarse grain extent file extent is mapped to a single allocation unit. With fine grain distribution, each grain is interleaved 128K across groups of 8 AUs.

                  Fine distribution breaks up large sized I/O operations, into multiple 128K I/O operations that can execute in parallel, which benefits sequential I/Os.
                  Coarse and fine grain attributes are pre-defined, as part of system templates, for all system related files; e.g., controlfiles, flashback logs and online redo log files are defined as fine grain, whereas, archive log files and datafiles are coarse.


                  Database Block Size
                  The Oracle blocksize effect the bit stream packet sizes and frequency during an I/O call. By Default Oracle Database use 8K block Size.
                  Currently we process more data than some time ago. So scarcely Oracle will read only one block at a time.


                  Storage Segment Size

                  The segment size is the maximum amount of data that the controller can write at once on one physical disk in a logical drive before writing to the next physical disk.
                  The choice of a segment size can have a major influence on performance in both IOPS and throughput. Small segment sizes increase the request rate (IOPS) by allowing multiple disk drives to respond to multiple requests. Large segment sizes increase the data transfer rate (Mbps) by allowing multiple disk drives to participate in one I/O request. Use a small segment size relative to the I/O size to increase sequential performance.

                  For database and data warehousing, block sizes between 32–128 KB have proven to be more effective.

                  Oracle can perform a full table scan over the entire disk drive. This is called a scattered read.
                  Oracle’s sequential data read is for accessing a single index entry or single piece of data. Use a small segment size for an OLTP with little or no need for read-ahead data. Use a large segment size for a Decision Support System ((DSS) environment where you will be doing full table scans throughout your data warehouse.

                  To know proper value for your particular situation you need to know how much I/O per Operation you have in Kbytes.

                  SEGMENT SIZE optimal be >= I/O per Operation (ASM)


                  Recomendation set Segment size = ~I/O request by ASM

                  Here are a few typical results of this choice:
                  Fewer disks used per request
                  Higher IOPS
                  Ideal for random I/O requests, such as the database file system

                  IBM recommend start with Segment Size 256 Kbytes because AU Size is 1024 Kbytes. i.e 1MB / four data drives = 256KB segment size



                  SIZE OF LUNS

                  I recommend creating LUNs on the size of up to 10% of the total size of the ARRAY. You can explore all available controllers performing balancing the LUNS on the controllers e.g ARRAY 1024GB use 10 LUNS of 100GB each. If you have 2 controllers, you can config five LUNS in each Controllers.

                  I recommend 10% based on my own experience, I do not mean that this is the optimal for your environment, each environment should be analyzed.
                  With 14 new disks in each new production EXP700, is there an optimal way in which to present them ASM i.e. just provide 14 new (mirrored pairs of) disks and let ASM stripe over all of them or define groups of sizes 4 and 8 and is this more optimal in any way (assume for the moment that we don't need to have a separate group for anything such as an FRA, etc.) i.e. is a group of 14 disks better or worse than groups of (8 and 4 and 2) disks or (8 and 6) disks?
                  I recomend you using Mirroring on Storage and ASM with External Redundancy.

                  Create Array:

                  OLTP envoriments:

                  DATA/INDEX : Raid 5 using feature "Enclose Loss Protection".


                  Remember set SEGMENT SIZE >= I/O per Operation (ASM).

                  If you do not have is information you can set with 128 kbytes and later modify reviewing the information provided in the ASM.

                  REDO LOG ONLINE/CONTROLFILE: RAID 10 using feature "Enclose Loss Protection". Diskgroup (DG_LOG). I recommend RAID 10 because Oracle never will read this data only Write.


                  FRA with RAID 10 is recommended but really expensive, you can create a RAID 5.

                  I recommend use "Failure groups" if you have two storage systems and for each storage you set a "Failure Group" that is good.


                  Check this...

                  http://levipereira.files.wordpress.com/2011/02/systems_storage_disk_pdf_oracle_ds4000_bp.pdf

                  http://levipereira.wordpress.com/2011/02/18/hard-drive-and-storage-expansion-enclosure-installation-and-migration-guide/

                  http://levipereira.wordpress.com/2011/02/18/best-practices-for-running-oracle-rac-database-with-asm-on-ibm-ds4800-storage-subsystems/


                  *ASM Technical Best Practices [ID 265633.1]*

                  https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=WHITE%20PAPER&id=265633.1


                  I Hope this help you.

                  Regards,
                  Levi Pereira

                  Edited by: Levi Pereira on Mar 18, 2013 10:45 AM
                  1 person found this helpful
                  • 6. Re: Oracle 10.2 ASM on AIX 5.3 compatibility with IBM DS4300 & EXP700 storage
                    ora_tech
                    Hi,

                    Excellent post, I appreciate :)

                    thanks,
                    X A H E E R
                    • 7. Re: Oracle 10.2 ASM on AIX 5.3 compatibility with IBM DS4300 & EXP700 storage
                      799157
                      Hi Levi

                      A fantastic amount of detail and info in your answer: many thanks for taking the time to help. It is going to take me a couple more reads.

                      One thing leapt out at me and that was the comment that hot spares aren't useable with ASM - that is a bit of a shame. But another read through first, I think, and a read of the other papers you reference.

                      cheers

                      Pete