Integrating etckeeper with Logentries & Chef

When working within a team to maintain system infrastructure, properly documenting and communicating changes made to configuration files within /etc is fundamental to preventing knowledge gaps throughout your team.

integrating-etckeeper-and-chef-with-le (1)

While version control tools like git are helpful in tracking standard changes to a code base, git doesn’t capture metadata important to /etc like permissions of /etc/shadow. To address this need, we’ve been exploring etckeeper – a small the version control application developed by Joey Hess (of Debian fame) for recording packaging installed or removed from /etc. While working with etckeeper, it became apparent that tracking changes over time in context of other events occurring within our systems would be useful and easily accomplished with Logentries.

etckeeper has a number of execution stages and provides a mechanism for hooking into each stage via a set of simple scripts. Using the Logentries API, you can easily send etckeeper metadata at etckeeper’s post-install and commit phases. All you’ll need to start with is a token-based log in your Logentries account.

During the post-install phase we use a hook stored in /etc/etckeeper/post-install.d/40send-to-logentries. You’ll need to insert the correct token into the following script.

#!/bin/sh
set -e

pl="/var/cache/etckeeper/packagelist"
dirty="/var/cache/etckeeper/dirty"
host=`hostname`
inet=`ip -4 address show dev eth0 | grep inet | awk '{print $2}'`
appname=etckeeper
NC_OPTS="-w 3"

token="TOKEN_FROM_LOGENTRIES"
API=api.logentries.com
PORT=10000

# run this if /etc is not clean
if etckeeper unclean; then
    message="Logging changes in /etc after $HIGHLEVEL_PACKAGE_MANAGER run"

    echo "$token appname=$appname host=$host action=notification stage='post-install' message='$message' inet='$inet'" | nc $NC_OPTS $API $PORT
    # list changed files
    git diff --name-status > $dirty
    while read -r line
    do
        f=$line
        echo "$token appname=$appname host=$host action=config stage='post-install' change='$f' inet='$inet'" | nc $NC_OPTS $API $PORT
    done < "$dirty" fi if [ -e $dirty ]; then     rm -f $dirty fi # Installed/removed packages if [ -e $pl.pre-install ]; then     message="Installed or removed packages"     echo "$token appname=$appname host=$host action=notification stage='post-install' message='$message' inet='$inet'" | nc $NC_OPTS $API $PORT     (     etckeeper list-installed | diff -U0 $pl.pre-install - | tail -n+4 | egrep '^[-+]' || true     ) > $pl.diff
    while read -r line
    do
        name=$line
        echo "$token appname=$appname host=$host action=package stage='post-install' change='$name' inet='$inet'" | nc $NC_OPTS $API $PORT
    done < "$pl.diff"
fi

if [ -e $pl.diff ]; then
    rm -f $pl.diff
fi

During the commit phase we execute this hook in /etc/etckeeper/commit.d/10send-to-logentries. Again in the following script you will need to insert the correct token.

#!/bin/sh
set -e

dirty="/var/cache/etckeeper/dirty"
host=`hostname`
inet=`ip -4 address show dev eth0 | grep inet | awk '{print $2}'`
appname=etckeeper
NC_OPTS="-w 3"

token="TOKEN_FROM_LOGENTRIES"
API=api.logentries.com
PORT=10000

# run this if /etc is not clean
if etckeeper unclean; then
    message="Logging changes in /etc after $HIGHLEVEL_PACKAGE_MANAGER run"

    echo "$token appname=$appname host=$host action=notification stage='commit' message='$message' inet='$inet'" | nc $NC_OPTS $API $PORT
    # list changed files
    git diff --name-status > $dirty
    while read -r line
    do
        f=$line
        echo "$token appname=$appname host=$host action=config stage='commit' change='$f' inet='$inet'" | nc $NC_OPTS $API $PORT
    done < "$dirty"
fi

if [ -e $dirty ]; then
    rm -f $dirty
fi

Once the above hooks are deployed to your hosts and can access api.logentries.com, any state changes and package state changes in /etc will be sent to Logentries in real-time.

...
appname=etckeeper host=x01 action=package change='-dstat 0.7.2-4' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=config change='Dbash_completion.d/darcs' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=package change='+dstat 0.7.2-4' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=package change='-dstat 0.7.2-4' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=package change='+socat 1.7.2.4-1' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=package change='-socat 1.7.2.4-1' inet='192.168.1.70/24'
appname=etckeeper host=x01 action=notification message='Installed or removed packages' inet='192.168.1.70/24'
...

Once your etckeeper metadata is being sent to Logentries, there are several useful things you can do, including:

  • Building alerts to notify teams of new changes
  • Creating custom tags to easily spot specific changes
  • Build dashboards to visualize config changes over time, compared to systems behavior

If you use Chef, we developed an etckeeper_wrapper cookbook which extends the etckeeper-cookbook, providing a number useful features for deploying etckeeper and a handler for the chef-client to execute etckeeper at every chef-client run. This provides a small safety net for the ops team to revert changes if things go wrong. The etckeeper_wrapper cookbook can be found here at https://github.com/jcftang/etckeeper_wrapper.

Posted in DevOps, Scripts

Leave a Reply