2 Replies Latest reply on May 13, 2011 9:11 AM by 821871

    Sun AVS4 with ZFS failover

      I'm testing to gain some experience with the Sun Availability suite on two hosts with a very simple setup:

      A primary host, called fokke:
      SunOS fokke 5.10 Generic_139555-08 sun4v sparc SUNW,SPARC-Enterprise-T5220

      and a secundairy, called sukke:
      SunOS sukke 5.10 Generic_139555-08 sun4v sparc SUNW,SPARC-Enterprise-T5220

      On both hosts, I'm using /dev/rdsk/c1t0d0s6 of about 80gb as the data volume, and /dev/rdsk/c1t0d0s7 of about 100mb as the bitmap volume.

      I have setup the config using the command on both hosts:
      sndradm -e fokke /dev/rdsk/c1t0d0s6 /dev/rdsk/c1t0d0s7 sukke /dev/rdsk/c1t0d0s6 /dev/rdsk/c1t0d0s7 ip async

      This all seems to work fine. After a sndradm -m I see data going, and dsstat -m sndr went from 100% to 0%.
      Next, I made a zpool on the data volume (I am aware it's only a single disk, but this is just a test setup):
      zpool create tpool /dev/dsk/c1t0d0s6

      This also worked fine. Next, I copied some files to the zpool, and waited till dsstat -m sndr showed 0%.
      Next, I put it in logging mode using sndradm -l to test a situation where we would need to switch over to the secundairy node.
      After sndradm -l, dsstat -m sndr showed the state as "L".
      I then imported the zpool on sukke using zpool import -f tpool without problems. I had it available, and could read from and write data to it.

      The problem now is returning from the slave host sukke to the primairy host fokke. I've tried several approaches but none seem to work.
      The first, what seemed to me to be the most obvious one, was to simply do a reverse update using sndradm -u -r.
      As you can not have both zpools active I first exported the zpool on sukke using zpool export tpool. This would not be a preferred situation in a production environment as it means service downtime. The whole reason we're using AVS is to have high availability, downtime while AVS copies several terabytes is not ideal. In any case, the approach didn't work. The commands completed fine, but after doing the reverse update I couldn't import the zpool on the primary host:

      pool: tpool
      id: 5341753121485047760
      state: UNAVAIL
      status: One or more devices contains corrupted data.
      action: The pool cannot be imported due to damaged devices or data.
      see: http://www.sun.com/msg/ZFS-8000-5E

      tpool UNAVAIL insufficient replicas
      c1t0d0s6 UNAVAIL corrupted data

      I tried the same mechanism but this time using a full reverse sync, but that didn't work either.
      I also tried using zpool destroy instead of zpool export and several other crazy ideas, but nothing seems to work.
      My guess is that ZFS and AVS are both making block-level changes which causes the zfs information on the disk that zpool needs to be able to do the import to fail, but I am in no way certain of this.

      At this moment, the only option that seems available, is to disable the AVS set, make the slave host sukke the new primary host, then recreating a new AVS set this time from sukke to fokke, so the other way around.
      The reason we wil be using asynchronous replication is because both nodes are in a different datacenter, and we prefer the primary host to be in one of them as that's the one where we have field engineers etc. on-site.

      I am wondering, is there no way around this? We simply would like to be able to switch from A to B, then make changes while it's active on B, then switch back to host A using B's data.

      Any help would be greatly appreciated,

        • 1. Re: Sun AVS4 with ZFS failover
          What about native ZFS features ZFS send/receive ? It's possible send only difference after last replication.

          • 2. Re: Sun AVS4 with ZFS failover
            As far as I know ZFS send/receive is snapshot based, so at best, when running from cron, it'll be done once a minute. That's not really replication, more of a very frequent backup, and once a minute is not frequent enough I feel. I like AVS because it works transparent, once it's set up it'll do it's thing and I only need to check dsstat every once in a while with a Nagios plugin.

            Besides, even if I were to switch to some alternative form of replication, I would like this to work, simply because I've set my mind to it...