Triple mirroring on FreeNAS

I just set up a triple mirror on my home NAS:

NAME              STATE     READ WRITE CKSUM
sto               ONLINE       0     0     0
 mirror-0         ONLINE       0     0     0
   gptid/de1e...  ONLINE       0     0     0
   gptid/de63...  ONLINE       0     0     0
   gptid/c8a3...  ONLINE       0     0     0  (resilvering)


Triple mirror!  Three drives serving up the same exact one drive's worth of data, what is this insanity?  Paranoid much?  No, not really, it's step 1 of an upgrade.  I've been over the recommended 80% usage on my primary zpool for a couple weeks now, but with no "day" job, didn't think purchasing new drives was the best idea.  Well, I got some good news the other day, and I promptly celebrated by buying the drives I needed.  They arrived over the weekend, so now it's time to start the upgrade.

My original configuration was (2) 4TB drives in a simple mirror.  Many other folks set up their zpools as a RAIDZ (ZFS equivalent of a RAID-5) or a RAIDZ2 (ZFS version of RAID-6 with two parity disks) in order to get the most out of their storage, but I decided to keep my configuration simple and go with mirroring for a couple reasons:

1. Performance.  Mirroring generally performs better than parity-based RAID setups as there's no math involved.  Does a home NAS need that much performance?  Probably not, but it's a nice perk.
2. Ease of upgrades.  This is the real driving concern.  One of the limitations of ZFS is that you can expand, but you can only expand with a vdev similar to what you had before - or you can start completely from scratch.  With a mirror, you have the simplest form of a vdev, with two disks, so each time I want to upgrade (and you know upgrades always happen) I can upgrade two disks at a time.  If I went with a basic 3-drive RAIDZ, I would have to buy 3 more drives to add on another 3-drive RAIDZ to the zpool.  If, like some of my friends, I ponied up for a 5 or 6 drive setup -- my next upgrade would be 5-6 drives at a time, and suddenly I'm looking for a case that can handle that many.  So, sticking with 2 drives in a mirror allows me to add another two drives each time I'm ready for an upgrade.  Yes, mirroring is the most "expensive" in terms of the amount of space that you get for the amount of disks you invest, but let's be honest, at this point, even WD Blacks are really pretty inexpensive for the amount of storage space that you get.

So, what's with the triple mirror?

Well, when I bought my first two drives, I got them at the same time.  It's entirely possible that they are from the same batch - which means that if there was some sort of defect in the batch, one drive might mean that the second drive might fail soon.  That's the Achilles' heel of mirroring - if two drives in one mirror fails, you lose data.  RAIDZ2 (or RAID-6) can lose two drives anywhere in the array and be fine, but if the right two drives in your mirror fail, then you're sunk.  It's back to backups - and you do have backups, right?

So, with two new drives coming in, what we have here is two that might also be from the same batch?  Whatever to do?  That's where the triple mirror comes in.

1. Add one new drive to the system, reboot, partition the drive.
   To keep track of which drive is which, the following commands are useful:
     gpart list ada0
     camcontrol identify ada0
2. Add that drive to the zpool via the "zpool attach" command, creating a triple mirror.
     zpool attach {pool} {existing disk} {new disk}
3. Wait for the resilvering (ZFS-speak for rebuilding a mirror, get it?) to complete.
     zpool status {pool}
   For my 4TB drive, I saw an initial prediction of 8 hours  - and it wound up taking only about 6. That's NOT bad, and one of the reasons that mirroring beats RAIDZ* setups.  Another nice perk of ZFS is that since the RAID is aware of the filesystem, replacing a huge disk with a small amount of data written to it will only require that the data is rewritten.  With a conventional RAID controller, it has no idea what data has been written, so has to rewrite the entire disk.
4. Remove one of the original drives from the mirror with zpool detach.
     zpool status {pool}
     zpool detach {pool} {disk}
5. Blank the ZFS config on that drive
   As it turns out, this step is not necessary.  Once you zpool detach the old drive, it's clear enough that FreeNAS doesn't complain when you add it back in.
6. Reinstall the old drive that was removed with the second new drive.
   Here's a chance to physically rearrange the drives if desired.  I put the first old/new pair in slots 1 and 2 in my NAS, giving them ada0 and ada1, so the next pair was ada2/3.  This isn't necessary, and since FreeNAS uses GPTID, the pools are unaffected.
7. Extend the zpool with those two drives in a second mirror.
   I did this using the FreeNAS GUI.
8. Profit!  Start filling up the now larger zpool.

Note that what I wind up with is a 2 mirror zpool in which each mirror has one new drive and one old drive.  Therefore, if there is a problem with either batch of drives, I'm more likely to not lose both of the drives from one mirror.