5 Replies Latest reply on May 2, 2013 9:00 PM by Cindys-Oracle

    how to repair a broken pool

      Hi all,

      I have the following problem with my zpool:
      Actually I had a pool with four disks within two mirrors. This is how it looked like before the problems begun:
      pool: space0
      state: DEGRADED
      status: One or more devices has been taken offline by the administrator.
      Sufficient replicas exist for the pool to continue functioning in a
      degraded state.
      action: Online the device using 'zpool online' or replace the device with
      'zpool replace'.
      scan: resilvered 352K in 0h0m with 0 errors on Thu May 2 20:02:24 2013

      space0 DEGRADED 0 0 0
      mirror-0 ONLINE 0 0 0
      c8t0d0p0 ONLINE 0 0 0
      c8t1d0s2 ONLINE 0 0 0
      mirror-1 DEGRADED 0 0 0
      c8t2d0s2 OFFLINE 0 0 0
      c8t3d0s2 ONLINE 0 0 0

      As you can see I set c8t2d0s2 offline at this point due to a hardware error. So, I wanted to replace this disk. But I accidentally didn't replaced the disk but add a new one. Beside this, and I have no idea how this happened, c8t3d0s2 gets also lost. So at the moment I'm not able to import the zpool anymore. But all disks are still present. However, by trying to import the pool it looks now as followed:

      pool: space0
      id: 3053988171767573704
      state: UNAVAIL
      status: The pool was last accessed by another system.
      action: The pool cannot be imported due to unavailable devices or data.
      see: http://support.oracle.com/msg/ZFS-8000-EY

      space0 UNAVAIL missing device
      mirror-0 ONLINE
      c8t0d0p0 ONLINE
      c8t1d0s2 ONLINE
      c8t3d0s2 ONLINE

      device details:

      missing-2 UNAVAIL corrupted data
      status: ZFS detected errors on this device.
      The device has bad label or disk contents.

      So I ask myself if there is ANY option to tell the zpool to attach/add the missing disks from mirror-2 again. The disks are still connected and they are still accessible but aren't recognized anymore.
        • 1. Re: how to repair a broken pool
          I don't see how this could happen but to me and maybe someone else can provide better input, it does not look like you added a replacement disk, it looks like your existing devices got mixed up. For example, c8t3d0s2 was in mirror-1 and now its in mirror-0. It is possible that c8t2d0s2 is now missing so it looks like the mirror-1 vdev is missing. Can you put the disks back to where they were originally and retry the pool import?

          A few more things:

          1. You don't need to offline a disk before you replace it.
          2. Your pools are mostly built on s2, which represents the entire disk so a better config is building your pool on the whole disks like this:

          # zpool create space-0 mirror c8t0d0 c8t1d0 mirror c8t2d0 c8t3d0

          3. You also have a c8t0d0p0 which isn't recommended but its not doing any harm at the moment.

          I think your best chance of recovery is to put the disks back to where they were originally. Then, re-attempt the import.

          Thanks, Cindy
          • 2. Re: how to repair a broken pool
            hi cindy,

            no, mirror-0 is still correct with c8t0d0p0 and c8t1d0s2. Within mirror-1 shold be c8t2d0s2 and c8t3d0s2. The disks are connected to the same port as they always were.
            Although a zpool import doesn't work: "cannot import 'space0': one or more devices is currently unavailable" . I guess thats because I set the disk offline before. And now, that I'm unable to import the pool to add or attach the missing disks again, I wonder if there is some other option setting the disks back into the pool without having the pool online...?
            • 3. Re: how to repair a broken pool
              One disk that is off lined in a mirrored pool should not cause the disk to disappear from the zpool status output. In addition, it should not cause the zpool import to fail because there should be enough remaining redundancy to make the pool available. So, off lining the disk should not have caused this failure. Is the c8t2d0s2 device still available in the format output?

              Thanks, Cindy
              • 4. Re: how to repair a broken pool
                the forum is miss-formating my posts. currently while trying to import the pool the output looks like this:
                space0 UNAVAIL missing device
                --mirror-0    ONLINE
                ----c8t0d0p0 ONLINE
                ----c8t1d0s2 ONLINE
                --c8t3d0s2    ONLINE

                What you can't see is that c8t3d0s2 isn't a disk within mirror-0, it's a additional disk within space0. That's wrong and was my mistake. In fact, it is also physically a different device that the one the pool used to know as c8t3d0s2.

                Actually what I'd need to do is remove the current c8t3d0s2 from the pool (it is a different device now connected on a different port) and add the original c8t3d0s2 disk.
                I hope I could explain it somewhat understandable :-)
                The question is how can I remove a disk from a pool which is offline and cannot be imported. And to answer your question: yes, all devices are still available on the format output.

                • 5. Re: how to repair a broken pool
                  I see now. I think you mentioned it at first but I didn't understand due to the formatting. I get it now.
                  The problem is that after you add a disk with zpool add, you can't remove it. I can't understand what
                  happened to mirror-1 and why c8t2d0s2 disappeared altogether. I understand why c3t8d0s2 is now
                  a standalone disk because you added it as a completely different disk.

                  I don't see how this config could be recoverable. If the pool was still imported, then you might be able
                  to attach a new disk to c8t3d0s2 to recreate the second mirror, but I would need to test this first.
                  Since pool is not importable then I don't see how that would be possible anyway.

                  Do you have backups of this pool's data?

                  Thanks, Cindy