Saturday, January 31, 2015

The meaning of the size of directories in long ls listing

One thing folks new to linux run into is what it means when they see a size associated with a directory in a long listing, and many assume that it represents the amount of space taken up by files in the directory - but that quickly becomes apparent that is not the case. So, to explain the directory size as it shows up in ls output, let's have an example:

24-rocket:~> mkdir derp
25-rocket:~> ls -l derp
total 0
26-rocket:~> ls -ld derp
drwxr-xr-x 2 cmh cmh 4096 Jan 30 17:44 derp

I've created a directory called "derp" and we can see that it's 4096 bytes - 4kb.

27-rocket:~> for x in {1..10000}; do touch derp/$x; done
28-rocket:~> ls derp | wc -l
10000

I now created 10,000 empty files in derp - none of them contain anything.

29-rocket:~> ls -ld derp
drwxr-xr-x 2 cmh cmh 155648 Jan 30 17:44 derp

Notice the size of derp is now larger - because derp contains the info for those files, such as filenames, inodes, etc. Interestingly enough, zero-byte files don't take up any disk space except their entry in the directory itself.

30-rocket:~> rm derp/????
31-rocket:~> ls derp | wc -l
1000

I just removed all of the four-digit files from derp, so that would be 1000-9999. We now have only 1000 files in derp.

32-rocket:~> ls -ld derp
drwxr-xr-x 2 cmh cmh 155648 Jan 30 17:45 derp

However, even though we've gotten rid of most of the files in the directory, it's the same size. Directories don't shrink. (At least not that I know of.)

One way to think of directories is that they're just files - special files that contain info about other files.That's why a zero byte file takes no space on the disk - it's just an entry in the directory file. Once you put even a single byte into a file, then there's disk space being used.

In order to get the amount of disk space used in a directory, you'll want to use the "du" command. To see the size of a single directory, run this command:

du -hs {directory}

In order to show the size of all subdirectories and files in the current directory, sorted by size, use:

du -ms * | sort -n

 Replacing the -h (human readable output) with -m (output in megabytes) makes all the numbers consistent so things sort properly. Otherwise, 900kb looks like more than 4tb - and that's not quite the case!


Thursday, January 15, 2015

Remote sudo with reasonably secure supplying of password

I've recently been exposed to some scripts where the approach to getting info requiring root access on a remote box was to run ssh via an expect script, then run a sudo su -, supply the password, and run commands. No checking of the output of commands was run, the expect script just waited for the very last character of the prompt to shovel in more shit.

It kinda made me feel like this.
Did it work? Sure... but I still didn't think this was a great idea for about as many reasons as I listed steps. Probably most of you would agree. Plus, the output was then everything you'd get from an interactive session, so they had to use a pipeline of greps and awks to get the data they wanted.

It hurt my soul to look at it. Not only was it sloppy and dangerous, it was just inefficient and ugly as shit. Even when the output was a single line of exactly what they wanted, they actually had to add a tag to that line so their grep could find it - and then strip the tag.


So I was curious - what's the best way to supply a password to a remote system when you have ssh access and sudo - but not NOPASSWD sudo?  Also, as a bonus, I want to keep the password out of plain visible text (especially in ps output) as much as possible. I never want my password in a command line, I never want it in a file, and I never want it appearing anywhere where it could possibly be read. (but I'm funny like that)

Here's what I came up with which might be useful for you.

user=cmh
host=myhost.example.com
read -s -p "Enter the password for $user@$host: " MYPASS
sn=$(echo -E $MYPASS | \
    ssh $user@$host 'sudo -Sp "" /usr/sbin/dmidecode -s system-serial-number')

First we set the username and remote hostname in vars. Then, the third line is a read command which suppresses echoing the input (-s) and specifies a nice prompt with the -p option, putting the input into a var called "MYPASS".

The last line assigns the output of a command substitution (the "$(...)" ) to a variable "sn". The substituted command echoes the value of MYPASS with escape chars disabled. (-E) It then pipes that password to the ssh which connects to the remote host, running the sudo command with the option to take the password via STDIN (-S) and to nullify the prompt (-p "") and running the command "/usr/sbin/dmidecode -s system-serial-number".

Obviously getting the remote system's serial number is just one thing, but it's a good example.

One sudo, one command run, no interactive shells, and it returns exactly what you want. Easy peasy, clean, and elegant.

Notes/Limitations/Dire warnings:
  • If you had to supply STDIN to the remote command, you might be out of luck, since you're supplying the password to sudo. Interestingly, though - this works:
    echo -e "$MYPASS\nderp" | \
        ssh $user@$host 'sudo -Sp "" /bin/bash -c "cat > /tmp/derpina"'

    My password goes to sudo - and then the remainder of the STDIN, in this case the word "derp" - goes to the bash command, which creates a file /tmp/derpina containing that text. This is starting to border on ugly, though, and if sudo doesn't want a password (your account has NOPASSWD set for the command, for example) that password goes right on through to the command... and that's horribad.
  • Lemme just reiterate what's said above - if sudo doesn't look for a password - even though you told it to use the -S option, it ignores it and passes that password right on through to the remote command. That could be suboptimal.
  • Careful with the quoting. In the example I used single quotes because I wasn't expanding any vars, but if I had to - supplying options, for instance - I'd have to use double quotes, at which point I'd have to change the sudo's "-p" option to use single quotes - or backslash escape the double quotes. Shell quoting isn't tricky if you understand it, but can be a beast if you don't. (learn your shell quoting rules)
  • Obviously the ssh works best if you have public key authentication set up. However, even if you don't, the pipeline doesn't interrupt standard SSH password input. I've tested that this still works if the remote system asks for a password. Yes, you then need to enter the password twice. (so, use public key auth) You might be able to use SSH_ASKPASS to supply the password, but that's another topic altogether.
  • You're storing your password in a shell variable. On the upside, even if you were to export the variable, it wouldn't show up in the /proc/{pid}/env for your shell - but it would for subshells. You'd need root or be the same user to read those anyway.
  • As bad as the script was - I did learn about dmidecode's -s option. Previously I'd been using dmidecode -t 1 and processing the output. This illustrates two things:
    • You can learn something useful from other folks, even if it's cleverly hidden amongst shitty work.
    • Manpages are your friend. Read them, and go back to them often, especially when you're scripting with those commands. Especially for commands you "know" really well.
  • You could skip the echo (and any issues with options) by using a here string:
       
    ssh $user@$remote .... <<< "$MYPASS"
    This would probably avoid potential problems with backslashes or other odd characters.
Other limitations? Any better ways that I'm missing?

Monday, February 24, 2014

Move large files with a progress bar

Sometimes I want to move around large files, and instead of just sitting there and wondering how they're going, I like to see a progress bar along the lines of an scp or rsync with the --progress option.  If you have the "pv" utility installed, this is quite easy.  The following command works with bash, ksh, or zsh:

for file in *; do echo $file; pv $file > /path/to/destination/$file && rm $file; done

Formatted for use in a script:

for file in *
do
    echo $file
    pv $file > /path/to/destination/$file && rm $file
done

Sunday, February 23, 2014

Triple mirroring on FreeNAS


I just set up a triple mirror on my home NAS:

NAME              STATE     READ WRITE CKSUM
sto               ONLINE       0     0     0
 mirror-0         ONLINE       0     0     0
   gptid/de1e...  ONLINE       0     0     0
   gptid/de63...  ONLINE       0     0     0
   gptid/c8a3...  ONLINE       0     0     0  (resilvering)



Triple mirror!  Three drives serving up the same exact one drive's worth of data, what is this insanity?  Paranoid much?  No, not really, it's step 1 of an upgrade.  I've been over the recommended 80% usage on my primary zpool for a couple weeks now, but with no "day" job, didn't think purchasing new drives was the best idea.  Well, I got some good news the other day, and I promptly celebrated by buying the drives I needed.  They arrived over the weekend, so now it's time to start the upgrade.

My original configuration was (2) 4TB drives in a simple mirror.  Many other folks set up their zpools as a RAIDZ (ZFS equivalent of a RAID-5) or a RAIDZ2 (ZFS version of RAID-6 with two parity disks) in order to get the most out of their storage, but I decided to keep my configuration simple and go with mirroring for a couple reasons:

  1. Performance.  Mirroring generally performs better than parity-based RAID setups as there's no math involved.  Does a home NAS need that much performance?  Probably not, but it's a nice perk.
  2. Ease of upgrades.  This is the real driving concern.  One of the limitations of ZFS is that you can expand, but you can only expand with a vdev similar to what you had before - or you can start completely from scratch.  With a mirror, you have the simplest form of a vdev, with two disks, so each time I want to upgrade (and you know upgrades always happen) I can upgrade two disks at a time.  If I went with a basic 3-drive RAIDZ, I would have to buy 3 more drives to add on another 3-drive RAIDZ to the zpool.  If, like some of my friends, I ponied up for a 5 or 6 drive setup -- my next upgrade would be 5-6 drives at a time, and suddenly I'm looking for a case that can handle that many.  So, sticking with 2 drives in a mirror allows me to add another two drives each time I'm ready for an upgrade.  Yes, mirroring is the most "expensive" in terms of the amount of space that you get for the amount of disks you invest, but let's be honest, at this point, even WD Blacks are really pretty inexpensive for the amount of storage space that you get.

So, what's with the triple mirror?

Well, when I bought my first two drives, I got them at the same time.  It's entirely possible that they are from the same batch - which means that if there was some sort of defect in the batch, one drive might mean that the second drive might fail soon.  That's the Achilles' heel of mirroring - if two drives in one mirror fails, you lose data.  RAIDZ2 (or RAID-6) can lose two drives anywhere in the array and be fine, but if the right two drives in your mirror fail, then you're sunk.  It's back to backups - and you do have backups, right?

So, with two new drives coming in, what we have here is two that might also be from the same batch?  Whatever to do?  That's where the triple mirror comes in.
  1. Add one new drive to the system, reboot, partition the drive.
    To keep track of which drive is which, the following commands are useful:
      gpart list ada0
      camcontrol identify ada0
  2. Add that drive to the zpool via the "zpool attach" command, creating a triple mirror.
      zpool attach {pool} {existing disk} {new disk}
  3. Wait for the resilvering (ZFS-speak for rebuilding a mirror, get it?) to complete.
      zpool status {pool}
    For my 4TB drive, I saw an initial prediction of 8 hours  - and it wound up taking only about 6. That's NOT bad, and one of the reasons that mirroring beats RAIDZ* setups.  Another nice perk of ZFS is that since the RAID is aware of the filesystem, replacing a huge disk with a small amount of data written to it will only require that the data is rewritten.  With a conventional RAID controller, it has no idea what data has been written, so has to rewrite the entire disk.
  4. Remove one of the original drives from the mirror with zpool detach.
      zpool status {pool}
      zpool detach {pool} {disk}
  5. Blank the ZFS config on that drive
    As it turns out, this step is not necessary.  Once you zpool detach the old drive, it's clear enough that FreeNAS doesn't complain when you add it back in.
  6. Reinstall the old drive that was removed with the second new drive.
    Here's a chance to physically rearrange the drives if desired.  I put the first old/new pair in slots 1 and 2 in my NAS, giving them ada0 and ada1, so the next pair was ada2/3.  This isn't necessary, and since FreeNAS uses GPTID, the pools are unaffected.
  7. Extend the zpool with those two drives in a second mirror.
    I did this using the FreeNAS GUI.
  8. Profit!  Start filling up the now larger zpool.
Note that what I wind up with is a 2 mirror zpool in which each mirror has one new drive and one old drive.  Therefore, if there is a problem with either batch of drives, I'm more likely to not lose both of the drives from one mirror.

Wednesday, February 19, 2014

Choosing the best OS for a simple Raspberry Pi server

I'm a big fan of FreeNAS.  I was first exposed to it while evaluating NAS (network attached storage) options at work.  I spent quite a bit of time with it, and the more I used it, the more I liked it.  While it's just a storage OS, the fact that it's based on FreeBSD gives us the ability to run fully functional FreeBSD jails, and this lead me to consider fully replacing my old linux-based home server with my FreeNAS server, using jails for the services outside of the FreeNAS functions, such as DNS, DHCP, mail (postfix) and web services.  This was all well and good until I had my DNS/DHCP server moved to a jail.  When I rebooted the FreeNAS system, I discovered that the system was coming up and looking for addresses before the jail had started - for example, the NTP servers.  While this wasn't a huge issue, it lead me to look at a really lightweight server outside the FreeNAS system for critical services such as DNS and DHCP.  I almost immediately settled on the Raspberry Pi, a simple system on a chip (SoC) which was inexpensive and used very little power.  Not only would it make a good little server, but it's a cool system to play with, so I wound up buying two - my "prod" DNS/DHCP server, and my "dev" server where I could try new stuff.

To be clear, yes, there are many other ways that I could have solved the problem, but here I'm more interested in trying something new and getting my learn on, and the Pi looked like it could be fun to play with.  It's a home network, where you should be free to try cool new stuff just because.

In my selection process for a server OS, I should also mention that I have the most experience with the RedHat derived OS's such as CentOS and Fedora, so I'm most comfortable working in them.  It's not a 100% requirement, but it's nice to have.

The distros I tried:
  • Raspbian
    This is the most common Pi OS, and as such is the best supported, so this is an excellent choice for a server OS as it'll be easy to keep the software up to date, and it's got a wide selection of software to choose from.
    • pros: the most common, the most well known and tested, the most well supported.  Hard to argue with that, so it's where I started.
    • cons: it's a full desktop OS and as such contains tons of unneeded software, and it's not RedHat based.
  • Pidora
    This is Fedora ported to the ARM architecture.  With my professional linux sysadmin background being heavily based in RedHat and Fedora, I was eager to give this one a try.  Reading reviews of it online, however, it seems that this isn't as well supported.
    • pros: Based on Fedora with which I'm more comfortable
    • cons: not as well supported, not reviewed well.  Had it downloaded and installed and it looked like it was based on F18 when 20 is current on the "mainstream" Fedora.
  • RedSleeve
    This one really caught my attention.  I've been running systems with RedHat for years, so I know it well and I like it.  RedSleeve is a port of RHEL to the ARM architecture, so would be perfect for a minimal server OS.
    • pros: an actual server OS, and a truly minimal install from the get-go.  Based on my favorite server OS, so I was immediately comfortable and had it singing happy tunes almost immediately.
    • cons: hand ported by a small group of folks, the latest version is 6.1 (CentOS and RHEL are quite a bit farther ahead) and the primary maintainer said that updates to 6.x will probably be less active as he'll be working on the 7.x release.  While I love the concept, I want something a bit closer to current.
  • FreeBSD link
    Another promising selection since I'm getting more and more familiar with it from my experience with FreeNAS.  Was originally unofficially supported, but is now part of newly released FreeBSD 10.  I was able to get an img file and put that onto a SD card, but when I booted it up, I discovered that it didn't have the pkgng package management kit installed.  It offered to install it, but couldn't find it.  I then had to go with the ports in order to install anything, and downloading and extracting the ports tree took several hours.  I was running on a Class 6 SD card, so that might contribute to the slowness, but it just seems to me that until pkgng is fully supported, FreeBSD really isn't ready for a proper server _quite_ yet - but hopefully it will be soon.
    • pros: another real server OS with a minimal install that looks like it's getting real attention.  
    • cons: not fully official with pkgng support yet - but as it gets more attention it's going to be a real option, I think.
  • MINImal raspBIAN link
    Based on the raspbian, but built from the ground up as a minimal install.
    • pros: minimal install from the ground up, latest packages available.
    • cons: hand built so might not be the latest kernel/etc
  • Raspbian Server Edition link
    Starts with the latest Raspbian, but runs a script to strip out all the unneeded packages
    • pros: based on the most common OS, starts with the latest version.
    • cons: assuming the maintainer of the scripts has the right packages to remove.
As excited as I was when I found Redsleeve, I've gotta say it doesn't look like it's getting the level of attention I'd want for a real server OS, where you want to stay current.  Pidora looks like it's aiming more for the desktop market, so doesn't look like that will be much of an option, either.  So right off the bat my desire for something RedHat derived looks to be unfulfilled.  No big worries, there's really nothing horribly different about the other OS's, and running something Debian based will give me more opportunity to whine about the aptitude tools, which I don't think are quite as good as yum.  Minibian seems really good but since it's custom built, might not be as up to date.  Haven't had a chance to try raspbian server edition yet, but it looks promising as it's a server-oriented minimal install, but what really has my attention is the FreeBSD 10.  I'll keep coming back to that because I have a soft spot in my heart (or my head) for running minimal servers on *BSD stuff harking back to my very first home servers running OpenBSD.  If they get the pkgng issues sorted out, that'll be my choice, but until then, it looks like raspbian really is the best choice for a little RasPi based home server.

Wednesday, October 17, 2012

Lance's Fisting Technique

Even with ice in his beard, his fingers are warm.
It's getting into the cold season, but that doesn't mean you should stop riding.   With the right equipment, you can continue riding all the way through the winter, and don't have to subject yourself to the horrors of indoor stationary riding.

That said, even with the proper equipment, many folks have trouble with cold hands and feet, myself included, even with thick, heavy gloves.  Thick gloves can make controlling the bike more difficult, and if even thick gloves aren't keeping your fingers warm, then the call of the indoor trainer may be getting stronger.  However, one of my riding buddies (Lance) has shared a trick with me that allows me to ride through the winter with nothing more than ordinary full-finger gloves!



 

Here's how it works:

  1. Start your cold ride.  You may be warm from the house (or car) but you were resting, so you're not "warmed up".  Ride for a little while until you start to warm up, you should be breathing hard.  This usually coincides with your fingers starting to get cold.  
  2. At this point, stop riding, and pull your fingers back into your glove, making a fist inside the glove.  The heat from your palm will warm your fingers.  (On the road or a non-challenging MTB trail, you may also be able to do this while still riding, but if you try it and hurt yourself, I will point and laugh.)  
  3. This shouldn't take long, just long enough to get your fingers to stop being really cold.  Don't stop long enough that you cool down, either.  It's just a quick stop, if you're stopped for more than a minute you're probably doing it wrong. 
  4. Once your fingers are warm-ish, you can put them back in the glove like normal, and resume riding.  
  5.  Your fingers should stay warm for the rest of the ride!
Since your core temp is up and you're properly warmed up, the blood is flowing, and it's carrying heat from the exercise.  Since your fingers aren't brutal cold anymore, the blood is able to get to them, and keep them warm.  I've been doing this for a couple years now, and have been able to ride through sub-freezing temps using the same full-finger gloves that I rode in the summer.  Last year I was halfway through the winter when I realized that the 661 gloves I was using had an open mesh on the side of the fingers.  Even with the open mesh (I could see the skin of my fingers!) my fingers never got cold past that initial warm-up period.

Give it a try!

Now for toes, this trick doesn't work as well -- you can't make a fist with your feet!  I'm still working on this one, but for me, the trick seems to be having shoes that are big enough to have thicker socks and still leave room for your feet.  That seems to be the key - if your feet are too snug in the shoes, you won't get circulation, and you'll have the same problem as your fingers above. If your shoes are not quite big enough, you can cheat it out a little bit by loosening the straps, especially the ones towards the front of the foot.  I've found that helps a bit.  Wind can still be an issue, and I've heard that clipless pedals can actually act as a heat sink - a larger aluminum clipless pedal can draw heat out of your foot.  I dunno how legit that claim is since there is a plastic sole, and usually, a plastic cleat between your foot and the pedal, but I do run thicker insoles in the winter.  The big thing is don't restrict your feet, and I often try to wiggle my toes to make sure they're still getting circulation.

Monday, August 27, 2012

Full list of the Strava segments in Nockamixon

Most of the Strava segments in Nockamixon are my work.  After seeing the awful mess that is segments in Wissahickon, I decided to create what I thought were the key segments for Nockamixon and hopefully that would prevent everyone and their mom creating new, conflicting segments.  The only problem is sometimes I have trouble keeping track of all of them!  Plus, there are a couple good segments that were created by other folks.  Just creating this post so that I have a list of all the Nox segments, and can refer to it if I need.

Full Lap Segments:

Nox Full Loop (S/F at Tower Rd trail split) (http://app.strava.com/segments/2214794)

Originally, someone else created "Full Nox Loop" and it was good, except Strava's segment matching is kinda crappy, and if you left out the Hammer trail, you'd get a match, which is why Bob Eichlin isn't the KOM.  Someone else created Nox short loop CCW which cuts off the Hammer trail, but this doesn't fix the Strava matching problem, and so still matches both types of laps.  Someone else created Nox Full loop S/F at lot entrance tower rd sign, but that one starts at the trailhead of the lot.  I'm not a fan of that, since there is a bunch of two-way traffic on this part, and as such I prefer to roll in on that bit of trail easy, and then get going once I hit the trail split about 1/4 mile from the parking lot.  One thing that would be nice to add is a segment that matches if folks start at the 563 entrance, but since I don't do that, I probably won't bother.

North segments:

The north section of the park includes the Cold Spot and Hammer trails.

Cold Spot CCW (http://app.strava.com/segments/1079674)
Cold Spot CW (http://app.strava.com/segments/1172770)

Hammer CCW (http://app.strava.com/segments/1079793)
Hammer CW (http://app.strava.com/segments/1292643)

Cold Spot and Hammer CCW (http://app.strava.com/segments/1172780)
needed - Cold Spot and Hammer CW ( )

Central segments:

Since these two loops (Haycock Run and EWR) are rarely ridden as loops, I split them into upper and lower segments to improve matches.

Haycock Run uphill (http://app.strava.com/segments/1267108)
Haycock Run downhill (http://app.strava.com/segments/2200638)
Haycock Lower SB (http://app.strava.com/segments/2118401)
Haycock Lower NB (http://app.strava.com/segments/2119306)

EWR lower (connector) NB (http://app.strava.com/segments/1079884)
EWR lower (connector) SB (http://app.strava.com/segments/1079866)
EWR upper NB (http://app.strava.com/segments/1204480)
EWR upper SB (http://app.strava.com/segments/1247461)

South segments:

Since this is a loop that starts and finishes in close to the same area, it matches in either direction, so there's no point for CW and CCW segments.


Jaywalk and South Park: (http://app.strava.com/segments/1181277)

These two are little mini-segments starting from South Park Rd and climbing away in each direction.
South Park Rd climb (http://app.strava.com/segments/1254148)
S Park to Dam (http://app.strava.com/segments/1886087)