Investigating I/O - iostat

vijaydba_appsdba

    We have I/O wait problem

     

    We have 2 node RAC - 11gr2 Running on RHEL-5.5

     

    iostat on one ASM disk, a part of DISKGROUP is always high.

     

    Six Disks for a Disk Group;

     

    Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn

    sdd              30.70         0.53         0.11   10397962    2097852

    sdf              20.51         0.34         0.09    6562480    1749392

    sdg              18.71         0.33         0.07    6360012    1378332

    sdh              19.17         0.32         0.06    6221567    1195633

    sdi              14.98         0.25         0.06    4947293    1170696

    sdk              93.16         1.62         0.36   31487863    6949274

     

    These are 6 disks- allocated to a diskgroup.

     

    Is this normal that i/o will be different on a different disks in asm diskgroup. Or do I have a problem with sdk disk.

     

    How this situation can be investigated further if needed?

     

    Thanks,

    Vijayaraghavan.K

      • 1. Re: Investigating I/O - iostat
        John Thorton

        vijaydba_appsdba wrote:

         

        We have I/O wait problem

         

        We have 2 node RAC - 11gr2 Running on RHEL-5.5

         

        iostat on one ASM disk, a part of DISKGROUP is always high.

         

        Six Disks for a Disk Group;

         

        Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn

        sdd              30.70         0.53         0.11   10397962    2097852

        sdf              20.51         0.34         0.09    6562480    1749392

        sdg              18.71         0.33         0.07    6360012    1378332

        sdh              19.17         0.32         0.06    6221567    1195633

        sdi              14.98         0.25         0.06    4947293    1170696

        sdk              93.16         1.62         0.36   31487863    6949274

         

        These are 6 disks- allocated to a diskgroup.

         

        Is this normal that i/o will be different on a different disks in asm diskgroup. Or do I have a problem with sdk disk.

         

        How this situation can be investigated further if needed?

         

        Thanks,

        Vijayaraghavan.K

        it appears you suffer from Compulsive Tuning Disorder.

        Different disks contain different objects which are accessed differently.

         

        Which hot objects reside on sdk?

         

        >We have I/O wait problem

        post SQL & results that show above is true.

        • 2. Re: Investigating I/O - iostat
          vijaydba_appsdba

          Hi John,

           

          I am happy with Compulsive Tuning Disorder.

           

          "Different disks contain different objects which are accessed differently."

           

          These are member disks of a DISKGROUP so I/O on each disks should be approximately same. Why the last one alone is so varying?

           

          "We have I/O wait problem" - I have confirmed from top and vmstat and wait of more than 20 % in OS level and bi and bo also high.

          - - With What else  can I confirm the same.


           

          Thanks,

          Vijay

          • 3. Re: Investigating I/O - iostat
            Mark D Powell

            vijaydba, I agree with John.  Also the iostat numbers you posted do not appear very large to begin with.  Were these taken during peak database load or during low load?  You should compare the posted numbers to ones you get during peak database load.

            - -

            If you have a database wait problem, what kind of wait?  For how long on average? And where is the wait being generated?  Those are the questions you need to ask and answer if the problem is Oracle related.

            - -

            HTH -- Mark D Powell --

            • 4. Re: Investigating I/O - iostat
              John Thorton

              vijaydba_appsdba wrote:

               

              Hi John,

               

              I am happy with Compulsive Tuning Disorder.

               

              "Different disks contain different objects which are accessed differently."

               

              These are member disks of a DISKGROUP so I/O on each disks should be approximately same. Why the last one alone is so varying?

               

              "We have I/O wait problem" - I have confirmed from top and vmstat and wait of more than 20 % in OS level and bi and bo also high.

              - - With What else  can I confirm the same.


               

              Thanks,

              Vijay

              at what value do bi & bo go from being OK to being high?

               

              The whole purpose of any RDBMS is return answers which reside on disk; which requires bi & bo to be non-zero.

              Any DB that does no IO is an idle & useless DB.

               

              >wait of more than 20 % in OS level

              post proof that 20% is anything more than your arbitrary "threshold".

               

              You still have not posted any proof that any problem really exists.

              • 5. Re: Investigating I/O - iostat
                jgarry

                Just thinking out loud here, not to discount anything John or Mark said.

                 

                Yes, one of the features of asm is balanced i/o, so it is a reasonable question to ask when one disk is getting more than 3 times the activity of the others.

                Hmmm...

                Rebalancing in progress?

                Need for rebalancing that's not done yet?

                Actual disk problem?

                Mirroring from problem with another disk?

                Unlikely but possible hot spot for one disk?

                 

                So, how exactly have you defined the asm?

                 

                Might be informative to use the asmcmd chkdg.

                • 6. Re: Investigating I/O - iostat
                  JohnWatson2

                  Do you have a voting file on that disk? They are not striped in the way that database files are.

                  • 7. Re: Investigating I/O - iostat
                    AJ

                    Could it be that your disks making up the diskgroup have various size?

                    Unevenly disks can cause disk space allocation issues and performance problems.. (you cannot even create a diskgroup in 12.1.0.2 (except external redundancy) with disks of different size anymore due to that).

                     

                     

                    AJ

                    1 位用户发现它有用
                    • 8. Re: Investigating I/O - iostat
                      vijaydba_appsdba

                      No voting file on that

                      • 9. Re: Investigating I/O - iostat
                        vijaydba_appsdba

                        Hi Mark,

                         

                        Yes we have uneven disks allocated. Out of 2 TB allocated in 6 disks 1 TB is on this DISK which has high I/O than others

                         

                        Thanks,

                        Vijay

                        • 10. Re: Investigating I/O - iostat
                          John Thorton

                          vijaydba_appsdba wrote:

                           

                          Hi Mark,

                           

                          Yes we have uneven disks allocated. Out of 2 TB allocated in 6 disks 1 TB is on this DISK which has high I/O than others

                           

                          Thanks,

                          Vijay

                          sum of other 5 disk is about equal to single 1TB disk values.

                          • 11. Re: Investigating I/O - iostat
                            1049595

                            Uneven disk size can cause high I/O in asm disk. Please check the document below.

                            Verifying I/O Activity Balance Across Disks in ASM

                            1 位用户发现它有用
                            • 12. Re: Investigating I/O - iostat
                              Mark D Powell

                              Vijay, as others have pointed out assigning unevenly sized disk to ASM often results in performance issues.  If the large ASM disk is a performance issue however really depends on how many physical disk are behind the logical disk assigned to ASM in each case.

                              - -

                              HTH -- Mark D Powell -- 

                              1 位用户发现它有用
                              • 13. Re: Investigating I/O - iostat
                                wblanchard

                                My answer is just going to be a restatement of the question posed by others here.  What disks are your DISKGROUP using?  If your disk group is spread out over all of the disks you've listed then it could be a slight concern.  If however your DISKGROUP (e.g. DATA) is just on the one disk then it isn't unusual to see much higher I/O ops on that disk.

                                • 14. Re: Investigating I/O - iostat
                                  vijaydba_appsdba

                                  Thanks all !!!!!