System Templating with Bundle

Table of Contents

Introduction and Goals
Template Layout
Pushing Changes
Limitations

Introduction and Goals

Long ago, I started managing my personal workstation using similar tools to those we used to manage our servers. (The first commit in the version control repository for my workstation template is dated September of 1998.) I've since followed Stanford's build system through a major OS change and several major changes to system management techniques, and I'm in the process of moving to Puppet, which we were using before I left. But currently I still use what Stanford used before Puppet: a tool called bundle.

I maintain all changes laid over the base operating system in a separate repository even for personal systems primarily so that I can recover my system configuration in the event of a hardware failure and so that I can build other, similar systems easily. I also want history and audit trail of changes to system configuration files, the ability to make temporary local modifications and then revert back to a known-good state, and an easy-to-explain environment for other people to use when I'm unavailable for some reason. In other words, most of the same needs as version control software fills for programming.

A second goal of a system template is to allow me to make changes without logging on to that system. The system on which I'm working has my local editor set up, is fast and responsive, and already has all of my credentials. A remote login, on the other hand, may be slow, tedious, and not as rich in environment. I want to be able to make changes locally and then push them to the system.

Template Layout

A system template consists, basically, of three things:

  1. A set of configuration files
  2. Instructions for how to install them
  3. Machinery to push those changes to the system

Note that this leaves out anything that isn't a configuration file. Following Stanford's server best practices, I try to package all compiled software and use package management capabilities to manage it. The technique described here doesn't cover package management and a few other operations; see the limitations section below.

The Files

I lay out all the files in a system template matching their installation location on the target server. In other words, the /etc/krb5.conf file for a system is located in etc/krb5.conf relative to the root of the template. A short script /usr/local/bin/reprepro-upload that I haven't packaged is in usr/local/bin/reprepro-upload. I do this even if it means creating a bunch of directories with only one file.

We experimented with different template layouts at Stanford over time, and this layout seemed to work the best. The nested directories can be annoying, but tab completion on the command line and in Emacs mostly takes care of that, and it's wonderful to not have to search for the file you want to update. It also lets you name all files in the template the same as you will name them on the system, without worries about ambiguity.

The only exception to this is that I normally omit the leading period of dotfiles, such as /root/.bashrc.

Put into the template all configuration files modified on the system unless the modification can be automated another way and there's some reason not to put the file in the template (such as because it's changed by other automated processes). Files that are executable on the system should be executable in the template. Otherwise, I make all files in the template mode 644 or 755 and control access, if needed, by locking off the top-level directory, since it makes for less strangeness when installing them on the system. There's nothing more annoying than something breaking because you accidentally installed a file readable only by root.

The Bundle

Many people have worked out systems where you check out a revision control repository directly over your system /etc directory or otherwise control configuration files in-place. This has significant advantages in cool, but requires that you represent permissions, ownership, and other sometimes-tricky bits directly in your revision control repository. There are tools for doing this, but I prefer adding a level of indirection and using a tool to apply the template to the system.

The tool I use for this is bundle. I use bundle for three reasons. First, was in heavy production use at Stanford for more than fourteen years, which means it's been fairly thoroughly debugged. Second, its bundle language is extremely simple, straightforward, easy to understand, and easy to read. Third, I maintained it, made extensive changes to it, and know how to fix it if it breaks. The third reason doesn't apply to other people, but the first two might.

A bundle file is a set of directives that bundle executes in order if and only if some action is needed to make the system match the template. The key feature of bundles is that they're idempotent: if you run bundle twice in a row, the second time it won't do anything. This means that you can see exactly what changes bundle will make, without noise from remaking changes that aren't actually changes. It also means bundle doesn't keep messing with the timestamps of your files.

The bundle file for a template is conventionally named setup.b (.b is the standard extension for bundle files) and located at the top level of the template. Conventionally, the bundle must be run from the top of the template and therefore can use relative paths for all of the files in the template.

The vast majority of lines in a bundle file are file commands, which install a specified file onto the system (creating parent directories as needed). There are usually also a handful of dir and link commands and occasionally a delete command. That's usually all that's necessary. bundle can do some other, more complicated or lower-level things, but usually addline or filter aren't necessary. Avoid complexity unless it's really necessary.

I conventionally start all bundle files with:

    owner=root
    group=0

The latter is a habit I got into when our Solaris and Linux systems had different names for group 0; these days, it's probably safe to just say group=root. This ensures that, by default, all files are installed owned by root:root. Exceptions should set owner or group on those individual lines. Since I keep all the files in my template with mode 644 or 755, I don't set a default mode and only override the mode variable for files with unusual permissions. If you use a different permission scheme, you may have to do something else. (If you set a global mode of 644, be careful of files that should be executable.)

bundle supports statement-modifier system commands, which are run only if the file is changed. This is the only time that you should use system, since otherwise it's run unconditionally each time the bundle runs. Statement-modifier system commands are very useful for restarting a service whose configuration file you just messed with. For example, nearly every bundle I write has the following command:

    file etc/inetd.conf               /etc/inetd.conf \
        system="/etc/init.d/openbsd-inetd restart"

or something very similar. This restarts inetd when I install a new /etc/inetd.conf. Another common pattern is to set a single system command for a whole set of files, possibly using the special bundle variable %D for the file that was installed:

    system="/usr/sbin/postmap %D"
    file etc/postfix/locals           /etc/postfix/locals
    file etc/postfix/relays           /etc/postfix/relays
    system=

This is the same as listing each system command on each line, but it looks neater and saves typing. Be sure to clear the system variable at the end of the block, though, or you'll be running postmap a lot and doing strange things to your system.

Pushing Changes

This brings us to pushing changes to the system itself. The basic process of applying the template the system is simple: change directories to the top of the template and run bundle setup.b as root. However, you have to get the template onto the system first.

The traditional way to handle this is to put the template in AFS (or some other shared file system, but AFS is my favorite). Then, to apply the template to the system, just log on, become root, get any necessary AFS tokens (I prefer my system templates not to be world-readable usually, just in case).

There are a few problems with this, though. First, while all of my systems get AFS eventually, they may not have it at first, particularly if I'm dealing with odd kernels such as Xen guests. Second, I'm not (yet at least) running my own AFS cell. Third, it requires forwarding my AFS credentials to the remote system, and I prefer to trust systems as little as possible just on general principles.

For a while, I rsync'd the repository onto the system and then applied changes from the local copy. This worked reasonably well, but now that I've switched all my repositories to Git, there's no reason not to let Git handle the push itself. So what I do now is create a remote named after the system:

    git remote add --mirror <system> ssh://<system>/root/template

and prior to the first push create the Git repository on the remote system:

    mkdir /root/template
    cd /root/template
    git init --shared

(and lock off the permissions on /root/template if desired). Be sure that you have a reasonable version of Git installed. git-core or git from lenny or later will work.

Then, I use the following shell script (called dist-bundle and kept in the root of the template) to push changes to the system:

    #!/bin/sh
    set -e
    env GIT_SSH='./rsh-wrapper' git push <system>
    rsh -l root -x <system> /root/bin/apply-template "$@"

/root/bin/apply-template is the following shell script, which updates the checkout (git push doesn't do that in the remote repository) and then runs bundle with the supplied options so that I can pass along a -n or -c to see what would change without making changes.

    #!/bin/sh
    set -e
    cd /root/template
    git reset --hard
    bundle "$@" setup.b 2>&1

I use Kerberos rsh to handle the connection and authentication and obtain root tickets before running dist-bundle (I have some shell aliases that make that quick and easy to do) so that I don't have to allow remote logins as root via ssh. That's why the GIT_SSH setting above; it fools Git into thinking rsh is ssh. rsh-wrapper is just:

    #!/bin/sh
    exec rsh -x -l root "$@"

You can use a similar technique to set special ssh options and force the remote user to be root even if you don't use Kerberos rsh.

One advantage of this technique is that the current bundle is always available in /root/template. If I'm working on the system and make some temporary changes and then want to put the system state back, I can always just cd to that directory and re-run the bundle. I can even commit changes locally and pull them back to remote repositories.

Limitations

The primary limitations of this system are the limitations of bundle: it's great at installing and manipulating files and pretty bad at anything else. The advantage of a more sophisticated and comprehensive system such as Puppet is that you can manage packages, users, and other system resources. But Puppet also brings a large complexity cost, where as bundle is very simple and quick for the sorts of small modifications that I make to my workstation and my personal servers.

I supplement this system with simple text notes on things that it doesn't deal with directly. For example, I usually maintain a file named PACKAGES at the root of the template that lists the packages I've installed on the system (directly, not indirectly via dependencies) in case I need to rebuild it. Getting this file started can be a tedious affair of going through a package list, stripping out the automatically installed packages, and trying to figure out what remaining packages you really care about. If you rebuild a system from scratch, though, it's easy to keep notes if you start right away.

Similarly, for all the other actions (creating users, running one-time setup commands, building an OpenAFS kernel module) that bundle can't easily do, I keep a NOTES file in the top of the template. It also contains the bootstrap details: the things I did to the system to get it to the point where I could start running bundle. Again, it's best to start this file during a clean system build so that you get everything from the beginning.

Last modified and spun 2015-11-01