This discussion is archived
7 Replies Latest reply: Jul 23, 2012 4:47 AM by Sebastian Solbach (DBA Community) RSS

Moving from external redundancy to nfs storage

Christian Newbie
Currently Being Moderated
Hey, I am running GI 11.2.0.3 on SLES11.
Currently my votingfiles are on one asm diskgroup called +OCR configured with external redundancy;

While doing some failure tests it showed up, that the shared storage is not available for more than 40 seconds.
Therefore clusterware stack is going down, due to no accessible votingfiles.

Hence, I have to move my votingfiles.
I created 3 virtual machines on 3 esx plattforms.
Each vm is presenting one votingfile via nfg as described in: http://www.oracle.com/technetwork/products/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf

So, which steps needs to be performed to switch from by diskdgroup +OCR to a new configuration with 3 votingfiles on 3 servers ?
Or can I stay with this diskgroup and added those 3 votingsfiles as quorum files ? That would be the easy.... But is it possible ?

Chris
  • 1. Re: Moving from external redundancy to nfs storage
    onedbguru Pro
    Currently Being Moderated
    Why not add additional voting files on another diskgroup? I am not a fan of NFS in a RAC environment for anything, but that is just me. What have you tried? what errors did you get?

    did you google search for
    crsctl add vote ???
  • 2. Re: Moving from external redundancy to nfs storage
    Christian Newbie
    Currently Being Moderated
    Hey, I added the quorum disk via nfs mount from a storage indepent server.

    But crsctl query css votedisk is NOT showing this nfs mount.

    In general:

    Is it possible to add a quorum when running with external redundancy ?
  • 3. Re: Moving from external redundancy to nfs storage
    Sebastian Solbach (DBA Community) Guru
    Currently Being Moderated
    Hi Christian,

    no. For multiple reasons:

    a.) A diskgroup will contain data (in your case OCR and SPFile of ASM instance) and Voting disk in the header.
    The Quorum failgroup will take the Voting disk header information, but cannot hold other data.
    b.) An external diskgroup does not know of failure groups, let alone quorum failgroup. (This concept is non existing because not needed).

    However for an external redundancy diskgroup, you can simply put one disk on NFS. This then will hold data + voting disks.

    And even if you have a normal redundancy diskgroup, you will always need at least 2 "normal" failgroups (for the normal files).

    Regards
    Sebastian
  • 4. Re: Moving from external redundancy to nfs storage
    Christian Newbie
    Currently Being Moderated
    Hey Sebastian,

    I am not really sure what you try to explain.

    My OCR Diskgroup is containing one single LUN from a storage.
    My current configuration (even when I am not sure that valid) is the OCR diskgroup with external redundancy and via asmca I added a nfs share as quorum disk. No error message appeared at this time.
    crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    -- ----- ----------------- --------- ---------
    1. ONLINE 43c2d2b6025b4f29bf3b1b4e87739de0 (ORCL:OCR) [OCR]
    Located 1 voting disk(s).
    ASMCA is still showing this configuration.

    SQL> select name,VOTING_FILE,REDUNDANCY from v$asm_disk;
    NAME V REDUNDA
    ------------------------------ - -------
    DATA N UNKNOWN
    OCR Y UNKNOWN
    RECO N UNKNOWN
    NFS03 N UNKNOWN
    SQL>

    When I get your right, I should use the created nfs disk and add it at normal disk to the diskgroup, right ?
    in detail:
    Diskgroup OCR will contain:
    LUN OCR external redundancy 5GB
    NFS Share external redundancy 5GB

    So my diskgroup OCR will contain 2 disks with a disksize of 10GB.

    Is that right ?
  • 5. Re: Moving from external redundancy to nfs storage
    Levi-Pereira Guru
    Currently Being Moderated
    Christian wrote:
    Hey, I am running GI 11.2.0.3 on SLES11.
    Currently my votingfiles are on one asm diskgroup called +OCR configured with external redundancy;

    While doing some failure tests it showed up, that the shared storage is not available for more than 40 seconds.
    Therefore clusterware stack is going down, due to no accessible votingfiles.

    Hence, I have to move my votingfiles.
    I created 3 virtual machines on 3 esx plattforms.
    Each vm is presenting one votingfile via nfg as described in: http://www.oracle.com/technetwork/products/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf

    So, which steps needs to be performed to switch from by diskdgroup +OCR to a new configuration with 3 votingfiles on 3 servers ?
    Or can I stay with this diskgroup and added those 3 votingsfiles as quorum files ? That would be the easy.... But is it possible ?
    Hi Chris,
    I do not understand what you are trying to do.
    You have a poor configuration placing voting disk on localdisk of each host. Correct if I'm wrong. This is what I understood. If this is true this configuration is not valid either for testing or educational environment.

    All files of Clusterware must be in a shared location out of host member of cluster. As a workaround you can create another NFS server (i.e New VM) that is not a member of the cluster for this purpose.

    About Quorum:
    To use a quorum failgroup is required only if you are using RAC Extended or if you are using more than 1 Storage in your cluster. Quorum ASMDISK is an inexpensive solution to solve problems of split-brain that apply on environment where it's possible e.g Extended RAC or if your cluster have even number of Storages (i.e 2,4 and so on).

    If you are trying just move voting disk from diskgroup with external redundancy to diskgroup with normal redundancy. This step is easy.

    Move Voting Disk to a temporary Diskgroup can be a existing Diskgroup or you can create a temporary diskgroup to hold it.
    So recreate your diskgroup with normal redundancy and move back your votedisk to this new diskgroup.

    Also you have a option of use only NFS to store votedisk. You don't need ASM (I don't recommend this for production environment).

    You can read it:
    http://levipereira.wordpress.com/2012/01/11/explaining-how-to-store-ocr-voting-disks-and-asm-spfile-on-asm-diskgroup-rac-or-rac-extended/

    Regards,
    Levi Pereira
  • 6. Re: Moving from external redundancy to nfs storage
    Christian Newbie
    Currently Being Moderated
    Hey Levi, sorry for my delayed answer.

    Currently I am running GI 11.2.0.3. OCR Diskgroup is configured with external redundancy.
    During failure test I discoverd that my diskgroup OCR is not available for nearly 40 secondes. So no access to votingfiles for 40secs. Therefore the cluster dies.
    No I need to find a solution for this scenario, where one of two technical rooms is isolated.
    I thought it my be a good idea to place a voting disk on a nfs share in a third technical room.
    So, will it work ? External redudancy with a quorum disk on a nfs share for the voting files ?
    Second idea. Create 3 nfs shares on 3 servers in 3 serverrooms. Create a new diskgroup for ocr and votingfiles. Replace existing diskgroup with the new one.

    Another way, would be to increase the timeout from nearly 20 seconds to 40 seconds. Is there a way of doing that ?

    Chris
  • 7. Re: Moving from external redundancy to nfs storage
    Sebastian Solbach (DBA Community) Guru
    Currently Being Moderated
    Hi Chris,

    a.) Voting Disk timout is normally 200 seconds ( crsctl get css disktimeout ). So this is not the direct error you see.
    However what might happen in your case is that due to a loss of access Diskgroup gets dismounted, which will then force the cluster to reboot, since it cannot access its clusterfiles. You may/can check this in the ASM alert.log

    b.) You cannot add a quorum failgroup to an external redundancy diskgroup. A external redundancy diskgroup does not know of failgroups, let alone quorum failgroups. So no this will not work.

    c.) Not really recommended. But would work. However I would create a new diskgroup on (the) 2 storage systems + one NFS share and move OCR, Vote (and ASM SPfile).

    d.) Timeout is high enough. You probably have a different issue (see a.)

    Regards
    Sebastian

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points