Posts for December 2011

2011-12-01: rra-c-util 3.11

This release fixes some problems with the new timeout functionality of the network utility library. When doing a non-blocking connect, the code to recover the errno from a failed connect was not working properly, leading to bogus error codes returned to the caller.

Also new in this release is a wrapper around some common APR include files that adds in some macros that weren't available in APR 0.9 (shipped with RHEL4).

You can get the latest version from the rra-c-util distribution page.

2011-12-02: WebAuth 4.0.2

This is an additional bug-fix release on top of the new 4.0 series. At this point, WebAuth should be stable and usable for the general community.

Bugs fixed in this release include one in cookie handling for REMOTE_USER preference cookies in WebLogin, compilation fixes without remctl and with older versions of APR (such as in RHEL4 and RHEL5), and improved documentation.

You can get the latest release from the official WebAuth site or from my WebAuth release page.

2011-12-06: Packaging the Sun Java JCE Policy

This is a less useful writeup now than it would be six months ago when I was first going to do it, but oh well. Some people are still stuck running the Sun Java JDK for a while longer, and maybe it will be useful.

Due to a variety of silly reasons too long to get into, the non-free Sun JDK comes with some encryption functionality disabled, particularly 256-bit AES. Those who want to use it have to download and install two replacement JAR files, which are not redistributable. While the Sun JDK is packaged for Debian in squeeze for those who have to use it (usually for legacy applications that haven't been ported to OpenJDK 7), the JCE policy files cannot be packaged.

The hardest part of this is that, due to how the Sun JDK is packaged and where the files have to be installed, it's tricky to install them without having to redo that work each time the Sun JDK is upgraded or reinstalled.

I wrote up instructions and packaging files for building packages that do the right thing, including across version upgrades of the Sun JDK. If you still have to use the Sun JDK, take a look. (And work on converting your software to work with OpenJDK 7 for the wheezy release!)

2011-12-11: spin 1.78 and spin-rss 1.24

I've been sitting on these minor updates to my web tools for a while.

The change to spin adds the \under command, which generates the HTML <u> tag. This is one of those things you're supposed to do with styles, but this was the easy solution for a journal post I wanted to make a while back.

The change to spin-rss fixes rendering into RSS of pages that end with lists instead of with text.

You can get the latest versions from my web tools distribution page.

2011-12-16: newsyslog 1.9

This is the first time I've had a chance to really work on this package in about six years, so much of what I did was haul it bodily up to the level of my current packages. It now uses Automake as well as Autoconf, uses the portability and utility library from rra-c-util, is closer to my current coding style, and has a more standard documentation layout. The test suite is still a bit bogus and has to be disabled if building outside of the source tree, but time enough to fix that another day.

The main point of the update is to add a couple of features that we've been missing. The main one is the ability to skew timestamps into the past for the purposes of determining the file and directory names to which logs are saved. We run newsyslog at 23:45 from cron.daily to try to rotate roughly a day of logs to the date corresponding to what's in the log, but since this is in cron.daily, sometimes other jobs are slow and push newsyslog later into the next day. When that happens, logs are saved under the wrong file names, conflict with the next day's logs, and cause other annoyances. The skew option can be used to generate dates as if it were some length of time earlier, adding more leeway for log rotation to run late.

Also added is the ability to name the same log for both copy and truncate. Previously, there was no way to both copy the current contents of the log and then truncate it (without creating a new file and renaming the old one). This is required for some broken software that can't cope with the log file being renamed but can cope with it being truncated, so this is now supported.

You can get the latest version from the newsyslog distribution page.

2011-12-24: Vacation status report

Stanford does a two-week mandatory shutdown over Christmas (this year slightly longer than that due to holiday placement), so I've been on vacation for about a week now, and I have about half of my vacation to go. I've unwound enough from work and started to get into the holiday mood of thinking over the past year and making tentative stabs at life organization for next year, so this seems like the time for a small bit of reflection.

First, it's amazing how much better I feel when I'm not conscious of being behind on routine tasks. I reached Inbox Zero on Monday after some concerted effort. This morning, I caught up completely on writing reviews for the first time in at least half a year and I think more than that. (I have them queued for posting over the next few days.) Both of those feel like huge milestones that have significantly improved my mood.

Second, I was doing better in concrete terms than I was doing in emotional terms. I was feeling overwhelmed and quite far behind, but I actually wasn't that bad off. One week of vacation has me completely caught up on the routine things that I've committed to. (There are other things I want to do and will if I have time, but those are outside the routine committments.) This implies that I was putting too much emotion, but almost enough work, into staying generally abreast of life.

Third, I've not been going into a deep flow state while working on things, and I've been missing that. Over the past couple of days, I decided what I most wanted to do is program on something that was entirely mine and that I could do properly, and decided to finish the test suite work for pam-krb5 that I've been working on off-and-on for about a year. I finished a nice, releasable chunk of work tonight and have mostly finished resynchronizing code between that project and C TAP Harness and rra-c-util, and am about in a position to do a release. I've been able to concentrate and do a huge amount of work in the past week: 59 commits and quite a lot of new code and complex debugging.

This qualifies as a great week of vacation. I'm feeling more relaxed, I've gotten a major chunk of work done, and I'm feeling in the mood to plan and rearrange and analyze life patterns to make a fresh start next year. I hope everyone reading this is having as good of a holiday season.

2011-12-24: C TAP Harness 1.9

This release adds a couple of new functions, test_tmpdir and test_tmpdir_free, to the basic TAP library. test_tmpdir function creates a temporary directory (tests/tmp) relative to the build tree and returns the path to that directory for test programs to use. test_tmpdir_free attempts to remove the directory and frees the path.

runtests has also been modified in this release to free all memory and resources it allocates, which makes it easier to run TAP test cases under valgrind without false positives.

You can get the latest version from the C TAP Harness distribution page.

2011-12-24: rra-c-util 4.0

The big addition in this release is a test driver framework for PAM modules that rests on top of the fake PAM library added in version 3.0. This is a data-driven test framework that allows most of a PAM test to be specified by a simple text configuration file. I'll write more about this later, including a complete example.

In support of that test framework, and in order to use both it and the fake PAM library with pam-krb5, the fake PAM library has been fleshed out considerably. It now handles everything that pam-krb5 uses, including authentication tokens and conversation functions, and supports intercepting getpwnam calls and returning a fake structure so that things involving ownership and home directories can be tested in isolation.

Also in this release are improvements to the PAM utility library, particularly around logging of PAM flags and avoiding memory leaks in option parsing.

In non-PAM news, this release avoids calling krb5_get_error_message with a NULL context, which older versions of Heimdal can't handle. The Kerberos probes can now cope with krb5/krb5.h being present and krb5.h missing (such as on NetDB), and there's a new Autoconf probe for Kerberos kadmin client libraries. That probe was written using a new, more generic framework for writing Autoconf library probes, to which the other probes will be converted when I find time.

There are substantial additions to the TAP add-on library, particularly around Kerberos. Test configuration is now expected to be in tests/config instead of tests/data, and temporary files are now created in tests/tmp instead of in the current working directory. There's a test function to get Kerberos password information from a configuration file, another to generate a krb5.conf file for testing, and another to determine the principal found in a keytab to avoid requiring the user to provide additional configuration giving the principal name. There's a new test function that uses the kadmin client library to set a password expired for test cases involving that. And, finally, there's a generic TAP add-on that wraps asprintf and vasprintf and checks for failure, calling bail if necessary.

Finally, the remctl Autoconf probe has been fixed (yet again) and now the optional version should work properly, and there are a few other, more minor bug fixes.

You can get the latest version from the rra-c-util distribution page.

2011-12-24: pam-krb5 4.5

The big news in this release isn't very user-visible, but it will be a huge help for further maintenance. pam-krb5 has finally been switched over to the generic PAM utility layer that I originally developed for pam-afs-session and now has a test suite that tests all the major functionality. It's not comprehensive by any stretch, but now I can add a few more tests with each release and slowly improve it. This release also imports the Kerberos portability layer from rra-c-util and uses it rather than rolling its own, which should produce better results.

In terms of user-visible changes, this release suppresses the password expired notice in the password stack if force_first_pass or use_first_pass are in use, since the output was otherwise confusing. It also checks to be sure it can obtain kadmin/changepw credentials before returning a password expiration error instead of incorrect password error to work around an issue with older Heimdal.

The location of the temporary root-owned ticket cache is now created relative to ccache_dir instead of forced to be in /tmp.

pam-krb5 now tells the MIT Kerberos library to prefer the older (and hence more compatible with older KDCs) change password protocol instead of the set password protocol. This isn't needed on Heimdal, since it's function always tries both.

There's a first pass at attempting to support the default OpenPAM build system, which makes all the module entry points static and instead exports a struct. I can't test, though, so I suspect this will need more work.

Logging has been improved, particularly around authorization checks and with defer_pwchange, and several minor memory leaks have been plugged.

Finally, there are some improvements and minor changes to the build system and other improvements for portability to BSD systems.

You can get the latest release from the pam-krb5 distribution page.

2011-12-25: git-pbuilder 1.25

As is typical, I've been sitting on several requests and updates for git-pbuilder for quite a few months. But the year-end holiday is always a good time to go through backlog and take care of things like that, when it's quiet and calm and there's a break from work.

This release adds support for traditional pbuilder as a possible builder program, which despite the name was not previously supported (at least since I converted an early verison of this script over to use cowbuilder). It alas cannot be determined from the name of the script for backward compatibility, since lots of cowbuilder users are invoking the script as git-pbuilder, but you can use real pbuilder by setting the BUILDER option. Thanks to Clint Adams for the initial patch.

While implementing that, I discovered that the code to set defaults for BUILDER, DIST, and ARCH from the program name (so that people can create symlinks like git-qemubuilder-squeeze-armel and have them do something appropriate) was pretty badly broken and was breaking the BUILDER environment handling. This should also now be fixed.

Finally, there's now a GIT_PBUILDER_AUTOCONF environment variable that can be set to "no" to tell git-pbuilder not to attempt to figure out any of the arguments to the builder. This is useful if you've already got complex logic in your .pbuilderrc to figure out the basepath/basetgz, architecture, distribution, and so forth and don't want git-pbuilder to bypass all of that. Note that this turns off everything, including --buildresult .., so if yoy use it, .pbuilderrc really has to handle everything.

You can get the latest version from my scripts distribution page, and I suspect it will show up in the git-buildpackage package before too long.

2011-12-25: Testing PAM modules

I maintain a couple of PAM modules, pam-afs-session and pam-krb5. I'm also a big fan of test-driven development and automated regression testing, and have been slowly converting all of my software over to using a test suite. But therein lies a challenge, since PAM modules are hard to test.

First, Linux PAM doesn't offer a good way to run a module in a test environment. Specifically, it doesn't offer a way to use an alternative PAM configuration other than the system PAM configuration, so that one can load only the module being tested (or it and some glue modules to set things up). One has to resort to dedicated test systems that do things like symlink part of their PAM configuration into a shared build tree.

Second, even if this problem were solved, Linux PAM doesn't (rightfully) allow a lot of introspection and digging around in its guts. This makes it harder to test whether data items, environment variables, and so forth are set properly and make sense. It also doesn't provide facilities for intercepting logging output that would otherwise go to syslog.

So, about a year and a half ago (and then with substantial improvements a year ago), I wrote a fake PAM library to test pam-afs-session. This library provides all of the PAM API that's called by a PAM module (at least the modules I'm testing), but has a much-simplified interface and allows test cases to look through its internals and be sure the module did things properly.

This was great as far as it goes. But writing the wrapper code to invoke a PAM module entry point and then check its return status, its logging, and so forth was complicated, messy, and difficult to understand. It also didn't handle testing prompting and other important aspects of PAM modules. So earlier this fall I wrote the beginnings of a PAM test driver that would move as much of the complexity as possible into shared code and into configuration files, and since then I've been expanding and improving it until it could handle the major testing required for pam-krb5 (which is much more complex than pam-afs-session). Yesterday, I released the results as part of rra-c-util.

Here's an example configuration file, this one for a pam-krb5 test that simulates prompting the user for a password and then obtaining credentials (but not creating a ticket cache):

        auth    = no_ccache
        account = no_ccache
        session = no_ccache

        authenticate  = PAM_SUCCESS
        acct_mgmt     = PAM_SUCCESS
        open_session  = PAM_SUCCESS
        close_session = PAM_SUCCESS

        echo_off = Password: |%p

        INFO user %u authenticated as %u

That's a complete and fairly complex test. The test framework takes care of running the listed entry points, passing them the given PAM configuration, and checking the return status. It provides a conversation function that checks the prompt and its type and replies with the text after |. As you can see, it supports some escapes that are replaced by data passed into the test suite driver, in this case the password to use for testing. And it checks the logging output, including the priority level, and similarly supports escapes there.

Running simple tests is very easy and is doable with a tiny C program. Here's a complete test program from pam-krb5, one that tests functionality that doesn't require a valid password and hence doesn't need to do much setup:

    #include <config.h>
    #include <portable/system.h>
    #include <tests/fakepam/script.h>

        struct script_config config;

        memset(&config, 0, sizeof(config));
        config.user = "root";
        run_script_dir("data/scripts/basic", &config);
        return 0;

That's all there is to it. That finds every configuration file like the above in the tests/data/scripts/basic directory in the package source and runs through each one, checking the results, and outputing the results as TAP output (the protocol used for the Perl test suite, and also my C TAP Harness package).

There are, of course, other things that it can do: more substitutions, setting the authtok in advance of calling the module, and a callback so that one can inspect PAM state before the PAM session is shut down. I've started writing some documentation, although so far it primarily covers the configuration file syntax and the headers are the best reference for the API.

The source code of the pam-krb5 package are the best reference so far. The library itself is part of rra-c-util and, like the rest of that package, is designed to be copied into the package that uses it and built alongside it so that it can be linked easily with the test programs.

It's worth noting that there isn't a way to use it to test installed PAM modules that are already bound to the system libpam library. The way one builds test programs currently is to link the test program with the individual objects making up the PAM module and then link with this library and the TAP library, since that ensures the right symbols are called. But it may be possible to do something tricky using LD_PRELOAD if anyone feels inspired.

2011-12-26: filter-syslog 2.0

filter-syslog is the tool that we (two groups at Stanford) use to mail any "interesting" lines from syslog to us each night. It runs as part of an analyze rule in newsyslog.

Some time back in internal discussions, the idea came up that it would be nice to filter out a range of lines, given a regex to match the start and another to match the end. This would let us filter out blocks of lines that are otherwise awkward to match, or whose specific details may change, like system reboots. At the time, I thought this would be easy. It turned out to require almost a complete rewrite of the state logic to deal with several of the tricky cases (seeing the range start but not the range end, several possible ranges that start with the same regex but have different ends, a range inside another possible range that fails). But I think I have all the cases taken care of now.

Separately, a co-worker had started using filter-syslog with raw regexes to filter Apache error logs, something that I thought should be easier. So I added support for parsing Apache error logs, stripping the timestamp and client IP, and matching them as a "program" of apache-level where level is the logging level.

These were big enough changes, particularly with the state rewrite, that it felt like it warranted a 2.0 release.

You can get the latest version from the filter-syslog distribution page. I'll upload new Debian packages to my personal repository, but probably not until tomorrow.

2011-12-29: rra-c-util 4.1

This release removes the Kerberos v4 portability code, which I've been carrying around but unable to test for some time. No one is shipping Kerberos v4 libraries any more, and hopefully few people are still using it. I'm releasing my last (non-obsolete) package that still supported Kerberos v4, and it seemed like a good time to drop it.

Also in this release is a replacement for a missing krb5_cc_get_full_name function (in Heimdal for some time and MIT only recently), and a bug fix in pam-util. The new pam-util option parsing code for handling Kerberos times didn't work properly on platforms where a krb5_deltat was not a long.

You can get the latest release from the rra-c-util distribution page.

2011-12-29: kstart 4.0

Between one thing and another, it had been almost two years since the last kstart release, which surprised me when I finally started working on it again. I'd been intending to put out a new release for some time, but apparently never got around to it. As a result, there were a ton of accumulated features and bug fixes, not to mention a huge update to the testing and portability framework.

The main backward-incompatible change in this release is that I finally dropped k4start from the distribution. I haven't been able to test it for years, no one (or almost no one) is shipping Kerberos v4 libraries any more, and I wanted to do a significant code restructuring. People who really need it can use older releases.

k5start and krenew now both allow arbitrary ticket cache designators to be passed with -k, rather than forcing the argument to -k to be a file cache, and both canonicalize the ticket cache name (by asking the Kerberos library for the real name) before passing it to subprocesses via the environment. These features combined should allow them to work much better with various non-file ticket caches.

k5start and krenew now both, when running as a daemon or when running a command, default to staying running even if authentication fails. This allows them to be more robust against temporary problems with contacting a Kerberos KDC, and is similar to what krenew -i previously did. krenew still exits by default if the ticket cache disappears or if the tickets are no longer renewable; to make it stay running in those situations, use krenew -i as before. Both k5start and krenew have a new -x option that restores the previous behavior of exiting on any error.

k5start, when run with the -o, -g, or -m options, now writes out a temporary ticket cache in the same directory, sets the ownership and permissions, and then does an atomic rename, closing a possible race where a process using that cache could temporarily not have access to it.

k5start and krenew both now propagate SIGINT to the child process when running a command rather than just exiting. Signal handlers are now set with sigaction, rather than signal, which will hopefully fix problems with propagating multiple signals.

The embedded kafs library has been updated to the current rra-c-util release, which adds support for Mac OS X and Solaris 11.

There are also a bunch of minor bug fixes and portability improvements, particularly to the build system.

You can get the latest release from the kstart distribution page.

Last modified and spun 2017-02-20