Monday, February 24, 2014

Move large files with a progress bar

Sometimes I want to move around large files, and instead of just sitting there and wondering how they're going, I like to see a progress bar along the lines of an scp or rsync with the --progress option.  If you have the "pv" utility installed, this is quite easy.  The following command works with bash, ksh, or zsh:

for file in *; do echo $file; pv $file > /path/to/destination/$file && rm $file; done

Formatted for use in a script:

for file in *
do
    echo $file
    pv $file > /path/to/destination/$file && rm $file
done

Sunday, February 23, 2014

Triple mirroring on FreeNAS


I just set up a triple mirror on my home NAS:

NAME              STATE     READ WRITE CKSUM
sto               ONLINE       0     0     0
 mirror-0         ONLINE       0     0     0
   gptid/de1e...  ONLINE       0     0     0
   gptid/de63...  ONLINE       0     0     0
   gptid/c8a3...  ONLINE       0     0     0  (resilvering)



Triple mirror!  Three drives serving up the same exact one drive's worth of data, what is this insanity?  Paranoid much?  No, not really, it's step 1 of an upgrade.  I've been over the recommended 80% usage on my primary zpool for a couple weeks now, but with no "day" job, didn't think purchasing new drives was the best idea.  Well, I got some good news the other day, and I promptly celebrated by buying the drives I needed.  They arrived over the weekend, so now it's time to start the upgrade.

My original configuration was (2) 4TB drives in a simple mirror.  Many other folks set up their zpools as a RAIDZ (ZFS equivalent of a RAID-5) or a RAIDZ2 (ZFS version of RAID-6 with two parity disks) in order to get the most out of their storage, but I decided to keep my configuration simple and go with mirroring for a couple reasons:

  1. Performance.  Mirroring generally performs better than parity-based RAID setups as there's no math involved.  Does a home NAS need that much performance?  Probably not, but it's a nice perk.
  2. Ease of upgrades.  This is the real driving concern.  One of the limitations of ZFS is that you can expand, but you can only expand with a vdev similar to what you had before - or you can start completely from scratch.  With a mirror, you have the simplest form of a vdev, with two disks, so each time I want to upgrade (and you know upgrades always happen) I can upgrade two disks at a time.  If I went with a basic 3-drive RAIDZ, I would have to buy 3 more drives to add on another 3-drive RAIDZ to the zpool.  If, like some of my friends, I ponied up for a 5 or 6 drive setup -- my next upgrade would be 5-6 drives at a time, and suddenly I'm looking for a case that can handle that many.  So, sticking with 2 drives in a mirror allows me to add another two drives each time I'm ready for an upgrade.  Yes, mirroring is the most "expensive" in terms of the amount of space that you get for the amount of disks you invest, but let's be honest, at this point, even WD Blacks are really pretty inexpensive for the amount of storage space that you get.

So, what's with the triple mirror?

Well, when I bought my first two drives, I got them at the same time.  It's entirely possible that they are from the same batch - which means that if there was some sort of defect in the batch, one drive might mean that the second drive might fail soon.  That's the Achilles' heel of mirroring - if two drives in one mirror fails, you lose data.  RAIDZ2 (or RAID-6) can lose two drives anywhere in the array and be fine, but if the right two drives in your mirror fail, then you're sunk.  It's back to backups - and you do have backups, right?

So, with two new drives coming in, what we have here is two that might also be from the same batch?  Whatever to do?  That's where the triple mirror comes in.
  1. Add one new drive to the system, reboot, partition the drive.
    To keep track of which drive is which, the following commands are useful:
      gpart list ada0
      camcontrol identify ada0
  2. Add that drive to the zpool via the "zpool attach" command, creating a triple mirror.
      zpool attach {pool} {existing disk} {new disk}
  3. Wait for the resilvering (ZFS-speak for rebuilding a mirror, get it?) to complete.
      zpool status {pool}
    For my 4TB drive, I saw an initial prediction of 8 hours  - and it wound up taking only about 6. That's NOT bad, and one of the reasons that mirroring beats RAIDZ* setups.  Another nice perk of ZFS is that since the RAID is aware of the filesystem, replacing a huge disk with a small amount of data written to it will only require that the data is rewritten.  With a conventional RAID controller, it has no idea what data has been written, so has to rewrite the entire disk.
  4. Remove one of the original drives from the mirror with zpool detach.
      zpool status {pool}
      zpool detach {pool} {disk}
  5. Blank the ZFS config on that drive
    As it turns out, this step is not necessary.  Once you zpool detach the old drive, it's clear enough that FreeNAS doesn't complain when you add it back in.
  6. Reinstall the old drive that was removed with the second new drive.
    Here's a chance to physically rearrange the drives if desired.  I put the first old/new pair in slots 1 and 2 in my NAS, giving them ada0 and ada1, so the next pair was ada2/3.  This isn't necessary, and since FreeNAS uses GPTID, the pools are unaffected.
  7. Extend the zpool with those two drives in a second mirror.
    I did this using the FreeNAS GUI.
  8. Profit!  Start filling up the now larger zpool.
Note that what I wind up with is a 2 mirror zpool in which each mirror has one new drive and one old drive.  Therefore, if there is a problem with either batch of drives, I'm more likely to not lose both of the drives from one mirror.

Wednesday, February 19, 2014

Choosing the best OS for a simple Raspberry Pi server

I'm a big fan of FreeNAS.  I was first exposed to it while evaluating NAS (network attached storage) options at work.  I spent quite a bit of time with it, and the more I used it, the more I liked it.  While it's just a storage OS, the fact that it's based on FreeBSD gives us the ability to run fully functional FreeBSD jails, and this lead me to consider fully replacing my old linux-based home server with my FreeNAS server, using jails for the services outside of the FreeNAS functions, such as DNS, DHCP, mail (postfix) and web services.  This was all well and good until I had my DNS/DHCP server moved to a jail.  When I rebooted the FreeNAS system, I discovered that the system was coming up and looking for addresses before the jail had started - for example, the NTP servers.  While this wasn't a huge issue, it lead me to look at a really lightweight server outside the FreeNAS system for critical services such as DNS and DHCP.  I almost immediately settled on the Raspberry Pi, a simple system on a chip (SoC) which was inexpensive and used very little power.  Not only would it make a good little server, but it's a cool system to play with, so I wound up buying two - my "prod" DNS/DHCP server, and my "dev" server where I could try new stuff.

To be clear, yes, there are many other ways that I could have solved the problem, but here I'm more interested in trying something new and getting my learn on, and the Pi looked like it could be fun to play with.  It's a home network, where you should be free to try cool new stuff just because.

In my selection process for a server OS, I should also mention that I have the most experience with the RedHat derived OS's such as CentOS and Fedora, so I'm most comfortable working in them.  It's not a 100% requirement, but it's nice to have.

The distros I tried:
  • Raspbian
    This is the most common Pi OS, and as such is the best supported, so this is an excellent choice for a server OS as it'll be easy to keep the software up to date, and it's got a wide selection of software to choose from.
    • pros: the most common, the most well known and tested, the most well supported.  Hard to argue with that, so it's where I started.
    • cons: it's a full desktop OS and as such contains tons of unneeded software, and it's not RedHat based.
  • Pidora
    This is Fedora ported to the ARM architecture.  With my professional linux sysadmin background being heavily based in RedHat and Fedora, I was eager to give this one a try.  Reading reviews of it online, however, it seems that this isn't as well supported.
    • pros: Based on Fedora with which I'm more comfortable
    • cons: not as well supported, not reviewed well.  Had it downloaded and installed and it looked like it was based on F18 when 20 is current on the "mainstream" Fedora.
  • RedSleeve
    This one really caught my attention.  I've been running systems with RedHat for years, so I know it well and I like it.  RedSleeve is a port of RHEL to the ARM architecture, so would be perfect for a minimal server OS.
    • pros: an actual server OS, and a truly minimal install from the get-go.  Based on my favorite server OS, so I was immediately comfortable and had it singing happy tunes almost immediately.
    • cons: hand ported by a small group of folks, the latest version is 6.1 (CentOS and RHEL are quite a bit farther ahead) and the primary maintainer said that updates to 6.x will probably be less active as he'll be working on the 7.x release.  While I love the concept, I want something a bit closer to current.
  • FreeBSD link
    Another promising selection since I'm getting more and more familiar with it from my experience with FreeNAS.  Was originally unofficially supported, but is now part of newly released FreeBSD 10.  I was able to get an img file and put that onto a SD card, but when I booted it up, I discovered that it didn't have the pkgng package management kit installed.  It offered to install it, but couldn't find it.  I then had to go with the ports in order to install anything, and downloading and extracting the ports tree took several hours.  I was running on a Class 6 SD card, so that might contribute to the slowness, but it just seems to me that until pkgng is fully supported, FreeBSD really isn't ready for a proper server _quite_ yet - but hopefully it will be soon.
    • pros: another real server OS with a minimal install that looks like it's getting real attention.  
    • cons: not fully official with pkgng support yet - but as it gets more attention it's going to be a real option, I think.
  • MINImal raspBIAN link
    Based on the raspbian, but built from the ground up as a minimal install.
    • pros: minimal install from the ground up, latest packages available.
    • cons: hand built so might not be the latest kernel/etc
  • Raspbian Server Edition link
    Starts with the latest Raspbian, but runs a script to strip out all the unneeded packages
    • pros: based on the most common OS, starts with the latest version.
    • cons: assuming the maintainer of the scripts has the right packages to remove.
As excited as I was when I found Redsleeve, I've gotta say it doesn't look like it's getting the level of attention I'd want for a real server OS, where you want to stay current.  Pidora looks like it's aiming more for the desktop market, so doesn't look like that will be much of an option, either.  So right off the bat my desire for something RedHat derived looks to be unfulfilled.  No big worries, there's really nothing horribly different about the other OS's, and running something Debian based will give me more opportunity to whine about the aptitude tools, which I don't think are quite as good as yum.  Minibian seems really good but since it's custom built, might not be as up to date.  Haven't had a chance to try raspbian server edition yet, but it looks promising as it's a server-oriented minimal install, but what really has my attention is the FreeBSD 10.  I'll keep coming back to that because I have a soft spot in my heart (or my head) for running minimal servers on *BSD stuff harking back to my very first home servers running OpenBSD.  If they get the pkgng issues sorted out, that'll be my choice, but until then, it looks like raspbian really is the best choice for a little RasPi based home server.