Saturday, August 22, 2015

Logging weather data with Raspberry Pi and collectd

I have a Raspberry Pi that I'm using to track temperature inside a house that isn't occupied year-round. I have a DS18B20 1-wire temperature sensor sticking out of the case of the Pi, but I also have it connecting to an Ambient Weather WS-1400-IP weather station, tracking the inside and outside temperatures. Initially, I was using curl with some bash tomfoolery to get the values, and alert if necessary. (inside temperature too cold, risk of pipes freezing, etc) This works, and works well, but recently I've been doing a bunch of work with Graphite and Collectd as part of a monitoring system at work. While Graphite is really, really awesome (I have several blog posts I want to do on it) I've been more and more impressed with the flexibility of Collectd. I've even got a Graphite server running at home with Collectd running on all my systems (except laptops, at the moment) just because it's cool and I'm a metrics nerd.

Well, I took another look at the Raspberry Pi that I have in this remote location, and decided that I wanted to set up a simple collectd to start logging some of the metrics on this system. Since I don't have a Graphite server there, I decided I'd just write the metrics to CSV files, so I'd have them, and if I needed, I could import those into my home Graphite server.

I started out with just some basic metrics, cpu, df, disk, load, memory, ntpd, ping, and swap. Got it up and running, looked good, started logging to the CSV files in my home directory. Cool. I then used the collectd APCUPSD plugin to grab metrics from the UPS that's connected to the system. Easy peasy.

Then I remembered coming across this awesome post a while back where the writer uses the collectd cURL plugin to read both CPU temps and connected 1-Wire sensors. I thought that was just cooler than the Poconos in the middle of February, so I had to go ahead and add the following config:

<Plugin curl>
  <Page "CpuTemp">
    URL "file:///sys/class/thermal/thermal_zone0/temp"
      <Match>
        Regex "([0-9]*)"
        DSType "GaugeLast"
        Type "temperature"
        Instance "CPUTemp"
      </Match>
  </Page>

  <Page "1WireTemp">
    URL "file:///sys/bus/w1/devices/28-0004330af2ff/w1_slave"
      <Match>
        Regex "(-+[0-9][0-9][0-9]+)"
        DSType "GaugeLast"
        Type "temperature"
        Instance "Room"
      </Match>
  </Page>
</Plugin>

I adjusted the file path to the 1Wire sensor, as the writer mentions "1wire-filesystem", which I don't use, so I just used the normal path to the 1-wire sensor. A couple small adjustments, and blammo, I'm logging CPU temperature and room temperature from the 1-wire sensor. Okay, granted - the 1-wire temperature is being written in millidegrees C, so 23 degrees C logs as "23000". At some point I'll discover how to change the data before it gets logged, but if I'm accessing this data in Graphite, I'll be able to use the wonderful Graphite functions (specifically, offset and scale) to convert it into whatever I want. It doesn't matter how I log it, just that I do.

If you're curious, the Graphite target for this conversion from millidegrees C to degrees F would be:

target=offset(scale(path.to.1wire.metric,0.0018),32)

That's 9/5, divided by 1000, and offset by 32 degrees. All calculated live on the Graphite server. Slick.

Then I realize -- this is the cURL plugin. The WS-1400-IP weather station has a base unit which makes the live weather data available on a web page, http://{weather station IP}/livedata.htm - why couldn't I just use the cURL plugin and hit that page?

Well, no reason aside from not being familiar with the cURL plugin - yet. Using the example above as a template, I started off with a simple config (added to the block above) to grab the indoor and outdoor temperatures:

  <Page "WeatherStation">
    URL "http://{weather station IP}/livedata.htm"
      <Match>
        Regex "inTemp.*value=\"(-+[0-9]+\.*[0-9]*)"
        DSType "GaugeLast"
        Type "temperature"
        Instance "insideTemp"
      </Match>
      <Match>
        Regex "outTemp.*value=\"(-+[0-9]+\.*[0-9]*)"
        DSType "GaugeLast"
        Type "temperature"
        Instance "outsideTemp"
      </Match>
  </Page>

Restarted Collectd, and what do you know - it's logging those temperatures! Well, that was easy!

Let me just explain how this works. I define a page "WeatherStation" which has the URL. Then, I define any number of matches against that one page (so I only have to load it once for all the metrics I'm collecting) to grab metrics. In the case of the inside temp, the output line in the HTML looks like this:

 <td bgcolor="#EDEFEF"><input name="inTemp" disabled="disabled" type="text" class="item_2" style="WIDTH: 80px" value="72.3" maxlength="5" /></td>

So, I write the regex to match against that line, starting with inTemp, then matching anything up to value=". Because the regex is surrounded by double-quotes, I had to escape the double quote in the regex with a backslash. I then put parenthesis around the actual metric that I want to extract. One thing to note is that I'm looking for zero or one "-" indicators (it gets cold up there) followed by one or more number, optionally followed by a literal "." (the decimal point) and more numbers. This allows my regex to match positive or negative integer or decimal values, which is important as the weather station reports temps to 0.1 degree, and humidity is logged as an integer. (percentage)

Starting with this template, it's only a moment of work to add the remaining metrics - inside and outside humidity, absolute and relative pressure, wind direction, speed and gust speed, solar radiation, UV value and UV index, as well as rainfall accumulations.

If you read the description on the weather station, you'll see that it can report all of these metrics to Weather Underground, and I do have it configured to do just that. That's all well and good, but sending all that data to them, I don't have access to it, and I can't graph it exactly the way that I want, but now that I'm logging this data, and I can get it into Graphite - I can.

2 comments:

WhyPi said...

I'll have to try this myself soon but https://collectd.org/wiki/index.php/Target:Scale
seems to be what you want ;)

cmh said...

Excellent link, thank you! I'm using the Graphite offset/scale functions to convert as I access the data, but have thought it'd be nice to report everything in consistent units. Right now passing data from the DS18B20 and a DHT22 and getting it from the weather station, and I may get temps in three separate units - F, C, and millideg C. Makes graphs a little more work.