This discussion is archived
12 Replies Latest reply: Mar 28, 2013 3:28 PM by georgehorlacher RSS

If my active fibre path becomes unavailable the node reboots automatically

georgehorlacher Newbie
Currently Being Moderated
2 Node Cluster on Oracle Linux 6.2

I have an 11.2.0.3 cluster RAC installed using ASM voting disks and an ASM disk hosting a database. I'm testing the failover for multipaths. I disable one path and check connectivity then re-enable it using a Cisco Fabric manager. On some paths - I'm assuming its the active path, the server almost immediately reboots after disabling the path (shutting down the port). I have 8 paths to each LUN and Multipath -ll shows output such as:

mpathf (26366623936333162) dm-15 WTSSR__B,INVICTA
size=2.1T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 1:0:0:7 sdf 8:80 active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 1:0:1:7 sdr 65:16 active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 1:0:2:7 sdad 65:208 active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 1:0:3:7 sdap 66:144 active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 2:0:0:7 sdbb 67:80 active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 2:0:1:7 sdbn 68:16 active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 2:0:2:7 sdbz 68:208 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 2:0:3:7 sdcl 69:144 active ready running

When I created the asm disks I ran: oracleasm createdisk BIGDISK /dev/dm-15
I'm not sure if that is a problem, but the database is running fine as is the cluster.

One other thing - Its booting from SAN and when I installed it found the multipath device and installed fine to the LUNs (both Nodes). I did have to edit the /etc/multipath.conf and comment out the blacklist options before multipath -ll would show all the paths. However I did not do anything else with that file. I did notice in some of my reading that possibly I should have in the multipath.conf file a wwid followed by an iSCSI unique identifier, but since multipath -ll is showing all these paths I have not manually made multipath updates to the config file.

I'm no sure whats wrong, if I configured something wrong like the multipath.conf needs more path information, or if there is a config setting to prevent the nodes from rebooting too quickly if a path is taken away?

Edited by: GeorgeHorlacher on Feb 5, 2013 3:59 PM
  • 1. Re: If my active fibre path becomes unavailable the node reboots automatically
    User328666 Newbie
    Currently Being Moderated
    Did you set the multipath.conf parameters to say "keep going on a path outage" such as this ...

    features "1 queue_if_no_path"
    no_path_retry 300
  • 2. Re: If my active fibre path becomes unavailable the node reboots automatically
    Sebastian Solbach (DBA Community) Guru
    Currently Being Moderated
    Hi,

    first of all try to isolate the problem a little. Since a server reboot could be caused by no access to the root disks and by the clusterware not able to access the votedisks.
    I would first disable the clusterstack (crsctl disable crs) and shut it down (crsctl stop crs).

    Then try the test (especially the "active" path again).
    If the server reboots, then there is an issue with your boot from SAN, that it is not using a DM device, but a single path device (which I assume).

    Checking internet I found:
    http://www.oracle-base.com/aggregator/2011/12/13/linux-multipathing-when-booting-from-san/

    Reading this, I assume that your are still booting from single path, and not multipath.

    Also make sure you use the recommended setting from your SAN storage vendor for multipath.conf. They know best how multipath should be configured.

    Side note on "queue is no path": I am personally no friend of "queue if no path", because I have seen whole clusters stops because of this:
    => Customer used "queue if nopath" on a boot from SAN environment. This will "hold" all I/Os till a path is back online.
    => The path to the root disk failed. As a result the whole system halted, because it could not write logfiles, waiting for the path to get online (not rebooting, because of queue if nopath).
    => Because of that the Database on the server also did "hold". Clusterware tried to kick in and kill the node
    => Node kill was not working, as the system waited for system I/O and did never get to the shutdown routine.
    => Databases on other node started to "stall" because they waited on blocks from the "halted" system....

    True with the new IPMI mechanism in 11gR2, this solution is solved differently, however I am still no fan of this feature.

    Regards
    Sebastian
  • 3. Re: If my active fibre path becomes unavailable the node reboots automatically
    georgehorlacher Newbie
    Currently Being Moderated
    OK, I've tried the
    features "1 queue_if_no_path"
    no_path_retry 300
    Added to my default settings in multipath.conf - might be a little better but still reboots under these circumstances:

    I turn off the active path of the server that is currently the SCAN VIP then it goes down (oraclea in this example)
    $ORACLE_HOME/bin/srvctl status scan
    SCAN VIP scan1 is enabled
    SCAN VIP scan1 is running on node oraclea

    He should have another path to failover to.

    I discovered I only had two voting disks (when I installed I had 3). I went ahead and added 3 more LUNs and now 3 voting disks show up when I run:
    $ORACLE_HOME/bin/crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    --
    1. ONLINE 357c29d11bcd4f89bfc7fbe68c109681 (/dev/oracleasm/disks/RAC3) [DATA]
    2. ONLINE 84a017f479984f4fbf5f9c4fde4a4d79 (/dev/oracleasm/disks/RAC1) [DATA]
    3. ONLINE 33685da5fe3b4f4bbfae241060d607c5 (/dev/oracleasm/disks/RAC4) [DATA]
    Located 3 voting disk(s).

    I looked at the Boot Image /etc/multipath.conf and found it did have the same multipath.conf file I started with - having an entry for the boot disk.

    I was concerned that it has 0 for uid and gid... That is the same in my file /etc/multipath.conf as I never changed it. OracleASM has not complained and I've run tests against the Database hosted by these disks so I don't think the rights are causing any problems but maybe they are?

    multipath {
    uid 0
    gid 0
    wwid "26134333763623535"
    mode 0600
    }

    Disabling any other active path does not reboot a server node, just the one running as the Master of the cluster - hosting the SCAN VIP.

    Any more ideas?
  • 4. Re: If my active fibre path becomes unavailable the node reboots automatically
    Sebastian Solbach (DBA Community) Guru
    Currently Being Moderated
    Hi,

    can you post the alert.log of the ASM instance as well as the alert<nodename>.log from the clusterware home?

    Did I understand this correctly, if you switch of the path on node oracleB (without the SCAN), then nothing happens?
    And I assume you only have 1 SCAN then, because normally you should have 3.

    PS: There is no master node. The only thing which is called master is a role a node can have, which allows the node to write the OCR (all others are only reading this).
    This role will then failover to the other node. And this must not be in correspondance with the SCAN. E.g. your SCAN may fail over to the other node (due to public network outage), but the OCR master role will not. But you are right, if you only have one SCAN, it is most likely that the one having the OCR write role, is also the node having the SCAN (since it is the first node which started ;)

    This smells a little like you found a bug... I can imagine what is happening, but it should not.

    Regards
    Sebastian
  • 5. Re: If my active fibre path becomes unavailable the node reboots automatically
    georgehorlacher Newbie
    Currently Being Moderated
    Thanks everyone for the help so far.

    I wanted to make it fail again and just now when I shutdown one port / path on the storage side (still had 7 other paths available) and both servers went down and rebooted. I've tested multipath successfully on this device with vmware and citrix so I know they failover OK on the storage side with other OS's. They came up OK on different paths but there is definitely something wrong with my multipath failovers in this cluster environment.

    ./11.2.0/grid/log/oraclea/alertoraclea.log
    [crsd(9265)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
    ]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/oraclea/crsd/crsd.log.
    2013-02-07 14:22:49.046
    [ohasd(5526)]CRS-2765:Resource 'ora.crsd' has failed on server 'oraclea'.
    2013-02-07 14:22:50.401
    [crsd(9278)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/oraclea/crsd/crsd.log.
    2013-02-07 14:22:50.409
    [crsd(9278)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
    ]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/oraclea/crsd/crsd.log.
    2013-02-07 14:22:51.076
    [ohasd(5526)]CRS-2765:Resource 'ora.crsd' has failed on server 'oraclea'.
    2013-02-07 14:22:52.434
    [crsd(9291)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/oraclea/crsd/crsd.log.
    2013-02-07 14:22:52.443
    [ohasd(5526)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    2013-02-07 14:23:23.395
    [cssd(5973)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/oracleasm/disks/RAC3 will be considered not functional in 99830 milliseconds
    2013-02-07 14:23:23.395
    [cssd(5973)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/oracleasm/disks/RAC1 will be considered not functional in 99730 milliseconds
    2013-02-07 14:23:23.395
    [cssd(5973)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/oracleasm/disks/RAC4 will be considered not functional in 99730 milliseconds
    2013-02-07 14:24:13.406
    [cssd(5973)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /dev/oracleasm/disks/RAC3 will be considered not functional in 49820 milliseconds
    2013-02-07 14:24:13.406
    [cssd(5973)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /dev/oracleasm/disks/RAC1 will be considered not functional in 49720 milliseconds
    2013-02-07 14:24:13.406
    [cssd(5973)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /dev/oracleasm/disks/RAC4 will be considered not functional in 49720 milliseconds
    [cssd(5973)]CRS-1613:No I/O has completed after 90% of the maximum interval. Voting file /dev/oracleasm/disks/RAC4 will be considered not functional in 19720 milliseconds
    2013-02-07 14:25:03.416
    [cssd(5973)]CRS-1604:CSSD voting file is offline: /dev/oracleasm/disks/RAC3; details at (:CSSNM00058:) in /u01/app/11.2.0/grid/log/oraclea/cssd/ocssd.log.
    [cssd(5973)]CRS-1606:The number of voting files available, 0, is less than the minimum number of voting files required, 2, resulting in CSSD termination to ensure data integrity; details at (:CSSNM00018:) in /u01/app/11.2.0/grid/log/oraclea/cssd/ocssd.log
    2013-02-07 14:25:03.417
    [cssd(5973)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/oraclea/cssd/ocssd.log
    2013-02-07 14:25:03.459
    [cssd(5973)]CRS-1652:Starting clean up of CRSD resources.
    2013-02-07 14:25:03.629
    [cssd(5973)]CRS-1653:The clean up of the CRSD resources failed.
    2013-02-07 14:25:04.504
    [ohasd(5526)]CRS-2765:Resource 'ora.ctssd' has failed on server 'oraclea'.
    2013-02-07 14:28:35.469
    [ohasd(5006)]CRS-2112:The OLR service started on node oraclea.


    /u01/app/11.2.0/grid/log/oraclea/cssd/ocssd.log
    2013-02-07 14:25:03.417: [    CSSD][4075226880](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 3 configured voting disks available, need 2
    2013-02-07 14:25:03.417: [    CSSD][4075226880]###################################
    2013-02-07 14:25:03.417: [    CSSD][4075226880]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
    2013-02-07 14:25:03.417: [    CSSD][4075226880]###################################
    2013-02-07 14:25:03.417: [    CSSD][4075226880](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
    2013-02-07 14:25:03.417: [    CSSD][4075226880]

    I'm hoping these are relevent, the alert logs are so big I need to learn to cycle them before generating the error so I could include them all.

    It looks to me like the voting disks lost their paths - all 3 voting disks at the same time and that caused the server to reboot. I will look into configuring more then 1 SCAN address if that would help. Yes I just have one SCAN and it's chosen by whichever server boots up first. Let me know if you have any ideas, Thanks!
  • 6. Re: If my active fibre path becomes unavailable the node reboots automatically
    Sebastian Solbach (DBA Community) Guru
    Currently Being Moderated
    Hi,

    it seems your Path Failover takes too long. Voting Disks have a timeout of 200 seconds.
    If your path does not failover in less, then node is rebooted.

    This definitely is a misconfiguration in your multipath.conf. Talk with your storage vendor about the correct settings please. They know what Path Checker should be used, and what are the optimal settings.
    After you have done this, and it is still failing, post your multipath.conf here, maybe we see the issue.

    Regards
    Sebastian
  • 7. Re: If my active fibre path becomes unavailable the node reboots automatically
    Dude! Guru
    Currently Being Moderated
    You might want to check cat /usr/share/doc/device-mapper-multipath*/mult*.defaults to see if your configuration is listed.
  • 8. Re: If my active fibre path becomes unavailable the node reboots automatically
    User328666 Newbie
    Currently Being Moderated
    Don't worry about uid, gid, and mode. These are deprecated in mpio 6.x. You probably set them in 5.x, but now you need to use udev rules or asmlib.
  • 9. Re: If my active fibre path becomes unavailable the node reboots automatically
    georgehorlacher Newbie
    Currently Being Moderated
    I think I ruled out multipath.conf as the problem. I was able to install the oracleASM Driver that allows you to mount a traditional volume for File I/O. Using a script to copy a lot of big files I can easily see which path the data is moving over. Turning off an active path on the non SCAN Node kills clustering communication but does not cause a reboot. Then I could test the two paths on that server and when I stop one of the paths it switches to the other path successfully, 3/4 times without even a file IO error from the interruption. I believe multipath is working as I see it switch automatically to the other path on file IO activity.

    Oracle Clustering seems to grab a path and hang on to it without failing over. Is there some configuration settings for the cluster services specifically with multipath for voting disks on ASM? After Re-enabling the path that killed the clustering services, it does not try to restart the cluster services again either. The Voting disks and boot from SAN may be locking the path for those disks somehow?
  • 10. Re: If my active fibre path becomes unavailable the node reboots automatically
    georgehorlacher Newbie
    Currently Being Moderated
    I was able to get back to this and do some more testing on it.

    I updated to a newer patch set for clustering: 16307750
    Changed the setting in multipath.conf for device to include:

    user_friendly_names yes
    features "1 queue_if_no_path"
    failback immediate
    no_path_retry 2

    With these two changes, removing a path no longer rebooted the server.
    However, the cluster services still do not reconnect over the other path.
    File I/O will go accross both paths and failover just fine.

    crsctl check cluster -all

    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4534: Cannot communicate with Event Manager

    --
    The work around seems to be to restart manually as everything can come up and work on the other path when booting.
    I tried some crsctl stop crs and that fails along with starting them up manually.
    Is there some failover configuration I've missed for Oracle Services?
  • 11. Re: If my active fibre path becomes unavailable the node reboots automatically
    Dude! Guru
    Currently Being Moderated
    ASMLib, if you use it, cannot handle to see the same disk twice. You may need to exclude devices from the ASM scan order to make sure to use the Multipath device and not the actual disk.

    Configuring Oracle ASMLib on Multipath Disks
    www.oracle.com/technetwork/topics/linux/multipath-097959.html

    Regardless of ASMLib, if you have separate disks on separate controller paths, you might want to verify to have each controller path in a separate ASM failure group, and not to set up failure groups that use different controller paths.
  • 12. Re: If my active fibre path becomes unavailable the node reboots automatically
    georgehorlacher Newbie
    Currently Being Moderated
    Dude, you where exactly right, thank you so much for the link and info about asm that I was missing. Your awesome!

    After setting up the correct settings in both Nodes /etc/sysconfig/oracleasm and rebooting them I was able to failover as much as desired without bringing the cluster down.

    /etc/sysconfig/oracleasm settings that make it work for me:

    # ORACLEASM_SCANORDER: Matching patterns to order disk scanning
    ORACLEASM_SCANORDER="dm"
    # ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
    ORACLEASM_SCANEXCLUDE="sd"

    It took me a while to figure out that by using device-mapper I had to use "dm" in the SCANORDER. multipath -ll shows all my devices as dm-1 through dm-15, so the "dm" picks them up nicely now.

    Once I setup this way though, no more cluster services going down or servers going down, so it had to have been using a disk with only one path before making it fail on active failovers.

    Great Solution!

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points