6 Replies Latest reply: Jul 2, 2013 12:33 AM by SAML. RSS

    What storage should we buy ?

    Kavanagh

      Platform: Oracle Enterprise Linux 6.3

      Grid Infrastructure version: 11.2.0.3

      RDBMS version : 11.2.0.3

      Hardware : Not yet decided (Either Dell or HP )

      ++++++++++++++++++++++++++++++++++++++++++++++++++++

       

      We have a fairly busy retail application and we are planning to create a RAC DB for this application. The maximum data size this application could ever reach is  5 TeraByte. Data will be moved/purged monthly and ideally it shouldn't cross 4TB at any given time.

       

      For performance, we are thinking of a mid-level Storage using Fiber Channel (not NAS). But we are not looking for an enterprise class SAN like Hitachi or EMC as our maximum data size is just 5TB. We are looking for an affordable storage which will be attached locally via Fiber Cable. So, it is technically not SAN. Right ?     Not sure what this is called .

       

      Any suggestions?

        • 1. Re: What storage should we buy ?
          Kavanagh

          I forgot to mark this as a question.

          As per the FAQ of  new forums, you'll have only 15 minutes from the time of first post to change a discussion to a Question.

          • 2. Re: What storage should we buy ?
            srsatya

            Hi,

             

            >>We are looking for an affordable storage which will be attached locally via Fiber Cable

            The cheapest is JBOD's, but with the DB size of 4TB, I think you may hit with performance issues if you dont go for SAN. SAN has lot of advantages like CACHE, Disk failure redundancy....., if your DB instance afford long down time or slow db performance, you can think of JBOD's, but again its not an good storage architecture.

            • 3. Re: What storage should we buy ?
              Billy~Verreynne

              The question is what storage can you afford in meeting your basic storage requirements?

               

              SANs are expensive. The prices we are paying for a single 8Gb port I personally find offensive. I am not a big fan of fibre channel technology (though we use it, successfully, for most of our RACs). The type of storage architecture I prefer is that used by Exadata Database Machine. It is Infiniband based. Seriously fast. Minimal latency. And very reasonably priced.

               

              You can buy this commercially. But these vendors are usually not in the database market specifically. But given that Oracle's own flagship h/w product uses this technology, is a clear statement about redundancy, performance, scalability and costs, of the best storage for RAC. (We have build our own Infiniband storage servers and fabric layer which we use for our RAC development cluster - capable of 48TB in total, it is a fraction of the cost for a SAN even a quarter of the size).

               

              My suggestion is to give very serious consideration to Oracle's Database Appliance. We have been building RACs for many years. It is not easy. There are always issues of one kind or another, from h/w firmware that fails under stress testing, to getting correct combinations of driver software, to discovering that a specific kernel version is not supported by ACFS, etc, etc. We've had even a case where vendor certified h/w failed stress testing.

               

              Database Appliance removes all of that - Oracle provides a RAC that consists of server nodes and storage and Interconnect and o/s. All tested. All working. All certified to work properly together. At a very reasonable price (compared to SAN, very cheap in pricing).

               

              No R&D needed to get your RAC installed, configured, and working. No risk in buying h/w that may just prove not to be robust enough for a RAC. (a simple thing like a poor quality network chipset, used to decrease total server model costs, could result in an epic fail when trying to use that server in a cluster - been there and have seen that too)

               

              In my view, and based on my RAC experience of the last 10 years, the most logical choice for a small RAC of a few TB, is Oracle's Database Appliance.

              • 4. Re: What storage should we buy ?
                Kavanagh

                Thank you for your valuable suggestion Billy.

                Our manager has already ordered 3 HP proliant servers (each with 132gb RAM with four 8-core CPUs ) . Not sure if Infiniband card (I think its called HCA) is available in these HP servers.

                 

                I need to google and find out if there are any reliable midrange storage vendors like Oracle(Sun) or IBM which offer Infiniband storage.

                 

                 

                Any idea what layer/protocol does Infiniband use ? (like TCP )  I couldn't find much info on this from googling although the following pdf gave some clues

                 

                http://www.mellanox.com/pdf/whitepapers/InfiniBandFAQ_FQ_100.pdf

                • 5. Re: What storage should we buy ?
                  Billy~Verreynne

                  Kavanagh wrote:

                  Our manager has already ordered 3 HP proliant servers (each with 132gb RAM with four 8-core CPUs ) .

                  Too bad.. but I have a 3 node RAC (recently build) using the same h/w.

                   

                  Why not Oracle Database Appliance? No. Idea. Ask management and ITS strategy why... and not a single sound and logical justification.. <sigh>

                   

                  Not sure if Infiniband card (I think its called HCA) is available in these HP servers.

                  Yes. HCA (Host Channel Adapters). I have both HBA (fibre) and HCA (Infiniband) cards in these servers.

                   

                  I need to google and find out if there are any reliable midrange storage vendors like Oracle(Sun) or IBM which offer Infiniband storage.

                  Awesome. There also is SSD storage available over Infiniband (IB). A bit pricey, but seriously fast.

                   

                   

                  Any idea what layer/protocol does Infiniband use ? (like TCP )

                  IP over IB (IPoIB) is supported. So you have the full IP stack, from TCP and UDP to ICMP.

                   

                  The HCAs are dual ports. So we configure 2 IB NICs and bond these (e.g. /dev/ib0 and /dev/ib1 as /dev/bond0). UDP is the default Interconnect protocol for Grid/RAC.

                   

                  Oracle however wrote an Open Source IB specific protocol called RDS (Reliable Datagram Sockets) to replace UDP when using IB. Easy to setup. According to Oracle presentations at the time it was done, it has a 50% lower CPU overhead than UDP with 50% less latency. We are using it on the new HP 3 node RAC.

                   

                  You also have a protocol called RDMA (Remote Direct Memory Access). This essentially allows one to share RAM across the IB fabric layer. The SRP (Scsi Rdma Protocol) sits on top of this and implements a SCSI protocol interface over RDMA. We are using this for our custom storage layer.

                   

                  Unsure exactly what Exadata uses, but it is quite likely based on RDMA.

                   

                  From a storage management perspective, whether using IB or Fibre as I/O fabric layer, is very similar ito LUNs, LUN masking, and so on.

                   

                  And as the IB fabric layer is 40Gb/s (dual 40Gb/s ports per server), the layer can carry both your Interconnect and I/O - as is the case with Exadata Database Machine. This is a lot cheaper (and less wiring and admin) than using GigEther (10Gb/s) for Interconnect, and HCA (8Gb/s Fibre) as the I/O fabric.

                   

                  Also take a look at http://top500.org/statistics/list/ for stats on Interconnect families/architectures. IB grew from something like 4% when we first start using it as RAC Interconnect, to over 40% today. So it is a prevalent technology amongst the 500 fastest and baddest computer clusters on this planet.

                  • 6. Re: What storage should we buy ?
                    SAML.

                    Hi,

                     

                    Apart from storage fabric, I think we should also talking about IOPS.  From the post, look like there is an existing system already running.  In this case, you can perhaps check what is the required IOPS and work with storage vendor to get that required IOPS.  Nowadays, we are lucky we can choose from different type of storage, SSD, SCSI, SATA, hybrid, tiering, etc.

                     

                    Cheers,

                    SAM L.