14 Replies Latest reply on Jul 7, 2016 3:57 PM by vijaydba_appsdba

    Investigating I/O - iostat

    vijaydba_appsdba

      We have I/O wait problem

       

      We have 2 node RAC - 11gr2 Running on RHEL-5.5

       

      iostat on one ASM disk, a part of DISKGROUP is always high.

       

      Six Disks for a Disk Group;

       

      Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn

      sdd              30.70         0.53         0.11   10397962    2097852

      sdf              20.51         0.34         0.09    6562480    1749392

      sdg              18.71         0.33         0.07    6360012    1378332

      sdh              19.17         0.32         0.06    6221567    1195633

      sdi              14.98         0.25         0.06    4947293    1170696

      sdk              93.16         1.62         0.36   31487863    6949274

       

      These are 6 disks- allocated to a diskgroup.

       

      Is this normal that i/o will be different on a different disks in asm diskgroup. Or do I have a problem with sdk disk.

       

      How this situation can be investigated further if needed?

       

      Thanks,

      Vijayaraghavan.K

        • 1. Re: Investigating I/O - iostat
          John Thorton

          vijaydba_appsdba wrote:

           

          We have I/O wait problem

           

          We have 2 node RAC - 11gr2 Running on RHEL-5.5

           

          iostat on one ASM disk, a part of DISKGROUP is always high.

           

          Six Disks for a Disk Group;

           

          Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn

          sdd              30.70         0.53         0.11   10397962    2097852

          sdf              20.51         0.34         0.09    6562480    1749392

          sdg              18.71         0.33         0.07    6360012    1378332

          sdh              19.17         0.32         0.06    6221567    1195633

          sdi              14.98         0.25         0.06    4947293    1170696

          sdk              93.16         1.62         0.36   31487863    6949274

           

          These are 6 disks- allocated to a diskgroup.

           

          Is this normal that i/o will be different on a different disks in asm diskgroup. Or do I have a problem with sdk disk.

           

          How this situation can be investigated further if needed?

           

          Thanks,

          Vijayaraghavan.K

          it appears you suffer from Compulsive Tuning Disorder.

          Different disks contain different objects which are accessed differently.

           

          Which hot objects reside on sdk?

           

          >We have I/O wait problem

          post SQL & results that show above is true.

          • 2. Re: Investigating I/O - iostat
            vijaydba_appsdba

            Hi John,

             

            I am happy with Compulsive Tuning Disorder.

             

            "Different disks contain different objects which are accessed differently."

             

            These are member disks of a DISKGROUP so I/O on each disks should be approximately same. Why the last one alone is so varying?

             

            "We have I/O wait problem" - I have confirmed from top and vmstat and wait of more than 20 % in OS level and bi and bo also high.

            - - With What else  can I confirm the same.


             

            Thanks,

            Vijay

            • 3. Re: Investigating I/O - iostat
              Mark D Powell

              vijaydba, I agree with John.  Also the iostat numbers you posted do not appear very large to begin with.  Were these taken during peak database load or during low load?  You should compare the posted numbers to ones you get during peak database load.

              - -

              If you have a database wait problem, what kind of wait?  For how long on average? And where is the wait being generated?  Those are the questions you need to ask and answer if the problem is Oracle related.

              - -

              HTH -- Mark D Powell --

              • 4. Re: Investigating I/O - iostat
                John Thorton

                vijaydba_appsdba wrote:

                 

                Hi John,

                 

                I am happy with Compulsive Tuning Disorder.

                 

                "Different disks contain different objects which are accessed differently."

                 

                These are member disks of a DISKGROUP so I/O on each disks should be approximately same. Why the last one alone is so varying?

                 

                "We have I/O wait problem" - I have confirmed from top and vmstat and wait of more than 20 % in OS level and bi and bo also high.

                - - With What else  can I confirm the same.


                 

                Thanks,

                Vijay

                at what value do bi & bo go from being OK to being high?

                 

                The whole purpose of any RDBMS is return answers which reside on disk; which requires bi & bo to be non-zero.

                Any DB that does no IO is an idle & useless DB.

                 

                >wait of more than 20 % in OS level

                post proof that 20% is anything more than your arbitrary "threshold".

                 

                You still have not posted any proof that any problem really exists.

                • 5. Re: Investigating I/O - iostat
                  jgarry

                  Just thinking out loud here, not to discount anything John or Mark said.

                   

                  Yes, one of the features of asm is balanced i/o, so it is a reasonable question to ask when one disk is getting more than 3 times the activity of the others.

                  Hmmm...

                  Rebalancing in progress?

                  Need for rebalancing that's not done yet?

                  Actual disk problem?

                  Mirroring from problem with another disk?

                  Unlikely but possible hot spot for one disk?

                   

                  So, how exactly have you defined the asm?

                   

                  Might be informative to use the asmcmd chkdg.

                  • 6. Re: Investigating I/O - iostat
                    JohnWatson2

                    Do you have a voting file on that disk? They are not striped in the way that database files are.

                    • 7. Re: Investigating I/O - iostat
                      AJ

                      Could it be that your disks making up the diskgroup have various size?

                      Unevenly disks can cause disk space allocation issues and performance problems.. (you cannot even create a diskgroup in 12.1.0.2 (except external redundancy) with disks of different size anymore due to that).

                       

                       

                      AJ

                      1 person found this helpful
                      • 8. Re: Investigating I/O - iostat
                        vijaydba_appsdba

                        No voting file on that

                        • 9. Re: Investigating I/O - iostat
                          vijaydba_appsdba

                          Hi Mark,

                           

                          Yes we have uneven disks allocated. Out of 2 TB allocated in 6 disks 1 TB is on this DISK which has high I/O than others

                           

                          Thanks,

                          Vijay

                          • 10. Re: Investigating I/O - iostat
                            John Thorton

                            vijaydba_appsdba wrote:

                             

                            Hi Mark,

                             

                            Yes we have uneven disks allocated. Out of 2 TB allocated in 6 disks 1 TB is on this DISK which has high I/O than others

                             

                            Thanks,

                            Vijay

                            sum of other 5 disk is about equal to single 1TB disk values.

                            • 11. Re: Investigating I/O - iostat
                              1049595

                              Uneven disk size can cause high I/O in asm disk. Please check the document below.

                              Verifying I/O Activity Balance Across Disks in ASM

                              1 person found this helpful
                              • 12. Re: Investigating I/O - iostat
                                Mark D Powell

                                Vijay, as others have pointed out assigning unevenly sized disk to ASM often results in performance issues.  If the large ASM disk is a performance issue however really depends on how many physical disk are behind the logical disk assigned to ASM in each case.

                                - -

                                HTH -- Mark D Powell -- 

                                1 person found this helpful
                                • 13. Re: Investigating I/O - iostat
                                  wblanchard

                                  My answer is just going to be a restatement of the question posed by others here.  What disks are your DISKGROUP using?  If your disk group is spread out over all of the disks you've listed then it could be a slight concern.  If however your DISKGROUP (e.g. DATA) is just on the one disk then it isn't unusual to see much higher I/O ops on that disk.

                                  • 14. Re: Investigating I/O - iostat
                                    vijaydba_appsdba

                                    Thanks all !!!!!