7 Replies Latest reply: Aug 11, 2010 12:10 PM by jablack RSS

    ASM Volumes on thin-provisioned SAN dirtying all blocks

    taupehat
      Hi there, sorry for the x-post from database-general but it was suggested that I do so. Anyhow, we've got 11g (11.1.0.7 with the 6851110 ASM patch recently applied) running on OEL 5 x86_64, with ASM connected to a raw, thin-provisioned ISCSI volume partitioned for DATA and FRA, and in every case where we do so, the SAN device reports within a few weeks that the whole volume has been allocated even though the DB (configured with autoextend on) is only holding about one tenth of the amount of available space on the device. What this means in systems terms is that somehow ASM is marking writes to nearly every block on the drive if only momentarily.

      In the original thread, there was speculation that a process of indexing AUs has led to the dirtying of the whole volume, but this would make more sense if the whole disk had been allocated immediately rather than over the course of a few weeks. My question is: what else could account for this behavior, and what steps can I take to help ensure that ASM behaves correctly in a thin-provisioned volume? (by "correctly" I mean writes contiguous blocks of data and doesn't dirty the whole thing)

      Thanks!
        • 1. Re: ASM Volumes on thin-provisioned SAN dirtying all blocks
          314801
          Hi,

          can you give us the reference to the original post please?

          Thanks in advance


          --
          Ronny Egner
          My Blog: http://blog.ronnyegner-consulting.de
          • 2. Re: ASM Volumes on thin-provisioned SAN dirtying all blocks
            762409
            Ronny,

            The original thread is : forums.oracle.com/forums/thread.jspa?threadID=1050763&tstart=0


            Mike,

            I have not tried ASM with thin provisioning myself, so will not try to answer as to why you are seeing what you are seeing. But I did get a couple of good documents detailing the usage of thin provisioning with ASM and they certainly do not describe what you are seeing. If you have not already seen these, maybe these will help you identify if there is something special happening on your environment.

            http://www.hds.com/assets/pdf/hitachi-dynamic-provisioning-software-best-practices-guide-oracle.pdf

            http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101586

            Definitely interested in knowing the answer when you resolve your issue, would appreciate and update accordingly.

            Thanks,
            Anindya
            • 3. Re: ASM Volumes on thin-provisioned SAN dirtying all blocks
              Bjoern Rost
              I find this question highly interesting. My first guess would have been that ASM does not neccessarily update blocks when it writes them but rather writes a new block and marks the old one as free. That could lead to the whole disk getting touched but I couldn't find anything that would prove this idea and from the whitepapers provided by the previous poster it looks like some other people have already done thin provisioning with success.

              So I have to ask this question: Are there any other files in your ASM groups other than datafiles? This could simply be archivelogs (or backups in the FRA) that are gathered (and kept) over time. But then again, this would be too simple =)

              Bjoern
              • 4. Re: ASM Volumes on thin-provisioned SAN dirtying all blocks
                taupehat
                Bjoern Rost wrote:
                I find this question highly interesting. My first guess would have been that ASM does not neccessarily update blocks when it writes them but rather writes a new block and marks the old one as free. That could lead to the whole disk getting touched but I couldn't find anything that would prove this idea and from the whitepapers provided by the previous poster it looks like some other people have already done thin provisioning with success.

                So I have to ask this question: Are there any other files in your ASM groups other than datafiles? This could simply be archivelogs (or backups in the FRA) that are gathered (and kept) over time. But then again, this would be too simple =)

                Bjoern
                Archivelogs are on, but that's accounted for in my enumeration of the volume usage, and they get wiped daily after backups. I'm going to dig through those whitepapers for any clues.
                • 5. Re: ASM Volumes on thin-provisioned SAN dirtying all blocks
                  taupehat
                  AnindyaR wrote:
                  http://www.hds.com/assets/pdf/hitachi-dynamic-provisioning-software-best-practices-guide-oracle.pdf

                  Definitely interested in knowing the answer when you resolve your issue, would appreciate and update accordingly.
                  That Hitachi document is a true gem - thank you greatly for posting it. I will conduct some experiments and update this thread with my results.
                  • 6. Re: ASM Volumes on thin-provisioned SAN dirtying all blocks
                    314801
                    Hi,

                    recently i had some time and did some tests with thin provisioning and ASM.

                    I used storage based on Opensolaris with ZFS thin provisioning against a 11g R2 database with 11g R2 ASM running on Windows. I created two LUNs and exported the LUNs via iSCSI. On the ASM side i formed a single disk group with external redundancy of the two LUNs presented and created one big file tablespace with approx 15 GB total size.

                    The storage systems shows the LUNs as follows:
                    NAME                       PROPERTY       VALUE    SOURCE
                    pool1/iscsi-racwin-temp05  volsize        15G      local
                    pool1/iscsi-racwin-temp05  usedbydataset  7.45G    -
                    pool1/iscsi-racwin-temp06  volsize        15G      local
                    pool1/iscsi-racwin-temp06  usedbydataset  7.45G    -
                    You can see: 15 GB total size reported while 7.45 GB are allocated. Thats pretty normal due to the data file created in the disk group.

                    During the night i ran a script which imported a schema and dropped it afterwards. The steps were repeated infinitely.

                    After more than 24 hours the thin provisioned disks look like this:
                    NAME                       PROPERTY       VALUE    SOURCE
                    pool1/iscsi-racwin-temp05  volsize        15G      local
                    pool1/iscsi-racwin-temp05  usedbydataset  7.47G    -
                    pool1/iscsi-racwin-temp06  volsize        15G      local
                    pool1/iscsi-racwin-temp06  usedbydataset  7.47G    -
                    As you can see there is a extremely small growth in size (from 7.45 GB to 7.47 GB). I observed this growth shortly after starting the very first import. Subsequent imports did not increased the actual allocated volume size.

                    So if we exclude the storage as a source for problems there might be the fact that 11g R1 ASM behaves different than 11g R2 ASM. I have not yet tested this...



                    --
                    Ronny Egner
                    My Blog: http://blog.ronnyegner-consulting.de
                    • 7. Re: ASM Volumes on thin-provisioned SAN dirtying all blocks
                      jablack
                      Hi Mike,

                      You mentioned you're structuring your asm diskgroups like Oracle recommends...a DATA and a FRA. For debugging purposes, if you could break out the components with more granularity, it may give you a little more insight into what's happening. (ie: separate data, temp, redo, backup diskgroups).

                      Has there been any resizing operations or asm_disk additions or removals?

                      This won't solve your issue, but maybe it'll compensate for it a bit...the ASRU utility lets the storage array know that storage is "reclaimable" for the array [<<see here>>|http://www.oracle.com/technology/products/database/asm/pdf/oracle_automatic_storage_management_and_thin_reclamation-1.pdf] so this gives you a way to "clean" the dirty blocks.

                      I hope that utility helps you.

                      Thanks,
                      John Black
                      Systems Engineer/Oracle DBA