11 Replies Latest reply: Jan 8, 2014 9:37 PM by Dude! RSS

Disk Rebalancing does not appear to resolve alert

lynwode Newbie
Currently Being Moderated

Happy New Year Folks!

 

Hoping someone has  a clear head today and can help answer the following.

 

Set up: Oracle 11g 2 node RAC on Windows 2008 running ASM

 

Problem:

An alert has been raised on both ASM instances as follows: Disk Group OCR_VOTE requires rebalance because the space usage imbalance between disks is high.

I have manually ran a rebalance on both instances but this has not cleared the alert. I have also tried to run the rebalance via EM but that doesn't work either.

 

I have connects as sys with SYSASM and run the follwing:

ALTER DISKGROUP OCR_VOTE REBALANCE POWER 5;

 

This appears to work as I get the output Diskgroup Altered but the alert remains.

 

Any ideas of what I am doing wrong or is this one of those annoying alerts that never clear.

 

Any help would be appreciated.

Cheers,

Tim.

  • 1. Re: Disk Rebalancing does not appear to resolve alert
    BillyVerreynne Oracle ACE
    Currently Being Moderated

    Explain the physical storage used - actual raw devices, sizes of raw devices, redundancy used, etc.

  • 2. Re: Disk Rebalancing does not appear to resolve alert
    lynwode Newbie
    Currently Being Moderated

    Hi Billy,

    The storage is SAN presented to Windows via fileshare (200GB - 147GB Free).

    Data 60GB - 8 Used

    FRA 55 GB - 6 used

    OCR_VOTE 30GB - 0.4 used


    All have External redundancy

     

    Hope this is what you are after?

  • 3. Re: Disk Rebalancing does not appear to resolve alert
    BillyVerreynne Oracle ACE
    Currently Being Moderated

    Not familiar with SANs usage by Windows. Do you by Fileshare mean SMB protocol between file server and database server?

     

    And how is it configured RAID/redundancy wise? Is  RAID used on the file server to provide a LUN via SMB that the Windows server use as a NTFS file system? Or are the raw disks on the file server presented as is (as a scsi device) to the Windows server (no RAID)?

     

    What does ASM use as raw devices? The actual SMB device? A "raw file" on the SMB mount/map drive? Something else?

     

    The ASM diskgroup for voting and OCR files are by default required as a high redundancy diskgroup (3 way "mirror"). One need to explicitly override this at install time if the LUN provided by the SAN is already redundant (RAID'ed) and external redundancy is used.

     

    How is your OCR_VOTE diskgroup configured (ito disks, failgroups and redundancy)?

     

    Are there any outstanding rebalancing processes in ASM (what does select * from v$asm_operation show)?

  • 4. Re: Disk Rebalancing does not appear to resolve alert
    lynwode Newbie
    Currently Being Moderated

    Hi

    I'm not familiar with Windows SAN usage either !

     

    I didn't set up the DB so I'll need to go back to the clients to answer the questions regarding the set up of LUNs and the OCR_VOTE diskgroup set up as EM is not giving me  a lot of information.

     

    I can confirm that there are no records in the v$asm_operation view - I have run the job via EM and via SQLPLUS connecting to both ASM instances.

     

    I'll try and report back later with more info.....

  • 5. Re: Disk Rebalancing does not appear to resolve alert
    lynwode Newbie
    Currently Being Moderated

    Some more info from the client:

    the san and currently uses a windows 7 machine using a shared folder. The asm also uses the san drive oracle partition in raw mode that it formats

     

    Not sure if this is of any further assistance - but any pointers would be helpful.

  • 6. Re: Disk Rebalancing does not appear to resolve alert
    BillyVerreynne Oracle ACE
    Currently Being Moderated

    Does not make much sense to me... Windows Server has a SAN s/w component  that supports LUNs, multiple I/O paths and so on - all the usual stuff one would expect with a SAN (running Fibre or Infiniband as I/O fabric layer). Why is this not used?

     

    It sounds, from your description, like a typical head-up-behind Windows-Duh! approach (hack) to storage.

     

    If the storage layer is reporting unexpected stuff (wrt to things like total size/available space), ASM could very well be confused. For example, if high redundancy is used for a diskgroup and the disks/LUNs/whatever used are reporting inconsistent/different sizes, it would want to balance the diskgroup.

     

    What is the output to the following sqlplus script?

    break on diskgroup skip 1
    select
            nvl(g.name, '<not mounted>')    as "DISKGROUP",
            d.name,
            d.path,
            d.mount_status,
            d.header_status,
            d.mode_status,
            d.state,
            d.redundancy,
            d.total_mb,
            d.free_mb,
            d.failgroup
    from    v$asm_disk d,
            v$asm_diskgroup g
    where   d.group_number = g.group_number (+ )
    and     'MOUNTED' = g.state (+ )
    order by
            g.name,
            d.failgroup,
            d.name
    /

  • 7. Re: Disk Rebalancing does not appear to resolve alert
    lynwode Newbie
    Currently Being Moderated

    Hi Billy,

    Unfortunately this is all the info I have in regards to the setup and the client does not have and design docs and I don't have the access/knowledge to go digging

     

    Here is the ouput of the query:

     

    OCR_VOTEOCR_VOTE_0000\\.\ORCLDISKOCR0CACHEDMEMBERONLINENORMALUNKNOWN1023710082OCR_VOTE_0000
    OCR_VOTEOCR_VOTE_0001\\.\ORCLDISKVOTING0CACHEDMEMBERONLINENORMALUNKNOWN2047720234OCR_VOTE_0001
    <not mounted>DATA_0000\\.\ORCLDISKDATA0CACHEDMEMBERONLINENORMALUNKNOWN6143752074DATA_0000
    <not mounted>FRA_0000\\.\ORCLDISKFRA0CACHEDMEMBERONLINENORMALUNKNOWN5631747303

    FRA_0000

  • 8. Re: Disk Rebalancing does not appear to resolve alert
    BillyVerreynne Oracle ACE
    Currently Being Moderated

    It seems like OCR_VOTE diskgroup is a normal redundant diskgroup with 2 fail groups? Or was it created with external redundancy?

     

    Either way, there is an inbalance as device ORCLDISKOCR0 is 10237MB in size, and device ORCLDISKVOTING0 is 20234MB in size. This means that irrespective of normal redundancy (fail group "mirrors"), or external redundancy (disks "striped"), there will be a problem due to the difference in size.

     

    ASM recommendations are disks of the same speed, and disks of the same size, in a diskgroup.

  • 9. Re: Disk Rebalancing does not appear to resolve alert
    lynwode Newbie
    Currently Being Moderated

    Thanks Billy,

    So are we saying that it is due to the underlying OS level file size and if these are the same the issue will be resolved?

  • 10. Re: Disk Rebalancing does not appear to resolve alert
    BillyVerreynne Oracle ACE
    Currently Being Moderated

    Possibly.

     

    ASM supports diskgroups consisting of different sized disks. The how depends however on the redundancy configured for that diskgroup.

     

    Read support note 460155.1 - it explains issues around freespace and extent allocation using disks with different capacity, and how ASM approaches the balancing issue on such diskgroups.

     

    I would however look at only using disks that are the same size in a diskgroup - as that is the recommendation.

  • 11. Re: Disk Rebalancing does not appear to resolve alert
    Dude! Guru
    Currently Being Moderated

    As far as I know an automatic disk rebalance occurs when the underlying disk configuration changes, in which case ASM needs to rebalance for performance and data redundancy reasons. If you use disks with different sizes then performance and use of free space will not be optimal.

     

    What I suspect in your situation is unreliable communication with your storage array, which could be a bad cable, bad storage controller, bad firmware, bad software driver or device. Or perhaps storage provide by your SAN is dynamic, which will confuse ASM.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points