2011-12-01: rra-c-util 3.11
This release fixes some problems with the new timeout functionality of the
network utility library. When doing a non-blocking connect, the code to
recover the errno from a failed connect was not working properly, leading
to bogus error codes returned to the caller.
Also new in this release is a wrapper around some common APR include files
that adds in some macros that weren't available in APR 0.9 (shipped with
RHEL4).
You can get the latest version from the rra-c-util distribution page.
2011-12-02: WebAuth 4.0.2
This is an additional bug-fix release on top of the new 4.0 series. At
this point, WebAuth should be stable and usable for the general community.
Bugs fixed in this release include one in cookie handling for REMOTE_USER
preference cookies in WebLogin, compilation fixes without remctl and with
older versions of APR (such as in RHEL4 and RHEL5), and improved
documentation.
You can get the latest release from the official WebAuth site or from my
WebAuth release page.
2011-12-06: Packaging the Sun Java JCE Policy
This is a less useful writeup now than it would be six months ago when I
was first going to do it, but oh well. Some people are still stuck
running the Sun Java JDK for a while longer, and maybe it will be useful.
Due to a variety of silly reasons too long to get into, the non-free Sun
JDK comes with some encryption functionality disabled, particularly
256-bit AES. Those who want to use it have to download and install two
replacement JAR files, which are not redistributable. While the Sun JDK
is packaged for Debian in squeeze for those who have to use it (usually
for legacy applications that haven't been ported to OpenJDK 7), the JCE
policy files cannot be packaged.
The hardest part of this is that, due to how the Sun JDK is packaged and
where the files have to be installed, it's tricky to install them without
having to redo that work each time the Sun JDK is upgraded or reinstalled.
I wrote up instructions and
packaging files for building packages that do the right thing, including
across version upgrades of the Sun JDK. If you still have to use the Sun
JDK, take a look. (And work on converting your software to work with
OpenJDK 7 for the wheezy release!)
2011-12-11: spin 1.78 and spin-rss 1.24
I've been sitting on these minor updates to my web tools for a while.
The change to spin adds the \under command, which generates the HTML <u>
tag. This is one of those things you're supposed to do with styles, but
this was the easy solution for a journal post I wanted to make a while
back.
The change to spin-rss fixes rendering into RSS of pages that end with
lists instead of with text.
You can get the latest versions from my web
tools distribution page.
2011-12-16: newsyslog 1.9
This is the first time I've had a chance to really work on this package in
about six years, so much of what I did was haul it bodily up to the level
of my current packages. It now uses Automake as well as Autoconf, uses
the portability and utility library from rra-c-util, is closer to my
current coding style, and has a more standard documentation layout. The
test suite is still a bit bogus and has to be disabled if building outside
of the source tree, but time enough to fix that another day.
The main point of the update is to add a couple of features that we've
been missing. The main one is the ability to skew timestamps into the
past for the purposes of determining the file and directory names to which
logs are saved. We run newsyslog at 23:45 from cron.daily to try to
rotate roughly a day of logs to the date corresponding to what's in the
log, but since this is in cron.daily, sometimes other jobs are slow and
push newsyslog later into the next day. When that happens, logs are saved
under the wrong file names, conflict with the next day's logs, and cause
other annoyances. The skew option can be used to generate dates as if it
were some length of time earlier, adding more leeway for log rotation to
run late.
Also added is the ability to name the same log for both copy and truncate.
Previously, there was no way to both copy the current contents of the log
and then truncate it (without creating a new file and renaming the old
one). This is required for some broken software that can't cope with the
log file being renamed but can cope with it being truncated, so this is
now supported.
You can get the latest version from the newsyslog distribution page.
2011-12-24: Vacation status report
Stanford does a two-week mandatory shutdown over Christmas (this year
slightly longer than that due to holiday placement), so I've been on
vacation for about a week now, and I have about half of my vacation to
go. I've unwound enough from work and started to get into the holiday
mood of thinking over the past year and making tentative stabs at life
organization for next year, so this seems like the time for a small bit of
reflection.
First, it's amazing how much better I feel when I'm not conscious of being
behind on routine tasks. I reached Inbox Zero on Monday after some concerted effort. This morning, I
caught up completely on writing reviews for the first time in at least
half a year and I think more than that. (I have them queued for posting
over the next few days.) Both of those feel like huge milestones that
have significantly improved my mood.
Second, I was doing better in concrete terms than I was doing in emotional
terms. I was feeling overwhelmed and quite far behind, but I actually
wasn't that bad off. One week of vacation has me completely caught up on
the routine things that I've committed to. (There are other things I want
to do and will if I have time, but those are outside the routine
committments.) This implies that I was putting too much emotion, but
almost enough work, into staying generally abreast of life.
Third, I've not been going into a deep flow state while working on things,
and I've been missing that. Over the past couple of days, I decided what
I most wanted to do is program on something that was entirely mine and
that I could do properly, and decided to finish the test suite work for
pam-krb5 that I've been working on
off-and-on for about a year. I finished a nice, releasable chunk of work
tonight and have mostly finished resynchronizing code between that project
and C TAP Harness and
rra-c-util, and am about in a position
to do a release. I've been able to concentrate and do a huge amount of
work in the past week: 59 commits and quite a lot of new code and complex
debugging.
This qualifies as a great week of vacation. I'm feeling more relaxed,
I've gotten a major chunk of work done, and I'm feeling in the mood to
plan and rearrange and analyze life patterns to make a fresh start next
year. I hope everyone reading this is having as good of a holiday season.
2011-12-24: C TAP Harness 1.9
This release adds a couple of new functions, test_tmpdir and
test_tmpdir_free, to the basic TAP library. test_tmpdir function creates
a temporary directory (tests/tmp) relative to the build tree and returns
the path to that directory for test programs to use. test_tmpdir_free
attempts to remove the directory and frees the path.
runtests has also been modified in this release to free all memory and
resources it allocates, which makes it easier to run TAP test cases under
valgrind without false positives.
You can get the latest version from the
C TAP Harness distribution page.
2011-12-24: rra-c-util 4.0
The big addition in this release is a test driver framework for PAM
modules that rests on top of the fake PAM library added in version 3.0.
This is a data-driven test framework that allows most of a PAM test to be
specified by a simple text configuration file. I'll write more about this
later, including a complete example.
In support of that test framework, and in order to use both it and the
fake PAM library with pam-krb5, the fake
PAM library has been fleshed out considerably. It now handles everything
that pam-krb5 uses, including authentication tokens and conversation
functions, and supports intercepting getpwnam calls and returning a fake
structure so that things involving ownership and home directories can be
tested in isolation.
Also in this release are improvements to the PAM utility library,
particularly around logging of PAM flags and avoiding memory leaks in
option parsing.
In non-PAM news, this release avoids calling krb5_get_error_message with a
NULL context, which older versions of Heimdal can't handle. The Kerberos
probes can now cope with krb5/krb5.h being present and krb5.h missing
(such as on NetDB), and there's a new Autoconf probe for Kerberos kadmin
client libraries. That probe was written using a new, more generic
framework for writing Autoconf library probes, to which the other probes
will be converted when I find time.
There are substantial additions to the TAP add-on library, particularly
around Kerberos. Test configuration is now expected to be in tests/config
instead of tests/data, and temporary files are now created in tests/tmp
instead of in the current working directory. There's a test function to
get Kerberos password information from a configuration file, another to
generate a krb5.conf file for testing, and another to determine the
principal found in a keytab to avoid requiring the user to provide
additional configuration giving the principal name. There's a new test
function that uses the kadmin client library to set a password expired for
test cases involving that. And, finally, there's a generic TAP add-on
that wraps asprintf and vasprintf and checks for failure, calling bail if
necessary.
Finally, the remctl Autoconf probe has been fixed (yet again) and now the
optional version should work properly, and there are a few other, more
minor bug fixes.
You can get the latest version from the rra-c-util distribution page.
2011-12-24: pam-krb5 4.5
The big news in this release isn't very user-visible, but it will be a
huge help for further maintenance. pam-krb5 has finally been switched
over to the generic PAM utility layer that I originally developed for
pam-afs-session and now has a test
suite that tests all the major functionality. It's not comprehensive by
any stretch, but now I can add a few more tests with each release and
slowly improve it. This release also imports the Kerberos portability
layer from rra-c-util and uses it
rather than rolling its own, which should produce better results.
In terms of user-visible changes, this release suppresses the password
expired notice in the password stack if force_first_pass or use_first_pass
are in use, since the output was otherwise confusing. It also checks to
be sure it can obtain kadmin/changepw credentials before returning a
password expiration error instead of incorrect password error to work
around an issue with older Heimdal.
The location of the temporary root-owned ticket cache is now created
relative to ccache_dir instead of forced to be in /tmp.
pam-krb5 now tells the MIT Kerberos library to prefer the older (and hence
more compatible with older KDCs) change password protocol instead of the
set password protocol. This isn't needed on Heimdal, since it's function
always tries both.
There's a first pass at attempting to support the default OpenPAM build
system, which makes all the module entry points static and instead exports
a struct. I can't test, though, so I suspect this will need more work.
Logging has been improved, particularly around authorization checks and
with defer_pwchange, and several minor memory leaks have been plugged.
Finally, there are some improvements and minor changes to the build system
and other improvements for portability to BSD systems.
You can get the latest release from the pam-krb5 distribution page.
2011-12-25: git-pbuilder 1.25
As is typical, I've been sitting on several requests and updates for
git-pbuilder for quite a few months. But the year-end holiday is always a
good time to go through backlog and take care of things like that, when
it's quiet and calm and there's a break from work.
This release adds support for traditional pbuilder as a possible builder
program, which despite the name was not previously supported (at least
since I converted an early verison of this script over to use cowbuilder).
It alas cannot be determined from the name of the script for backward
compatibility, since lots of cowbuilder users are invoking the script as
git-pbuilder, but you can use real pbuilder by setting the BUILDER option.
Thanks to Clint Adams for the initial patch.
While implementing that, I discovered that the code to set defaults for
BUILDER, DIST, and ARCH from the program name (so that people can create
symlinks like git-qemubuilder-squeeze-armel and have them do something
appropriate) was pretty badly broken and was breaking the BUILDER
environment handling. This should also now be fixed.
Finally, there's now a GIT_PBUILDER_AUTOCONF environment variable that can
be set to "no" to tell git-pbuilder not to attempt to figure out any of
the arguments to the builder. This is useful if you've already got
complex logic in your .pbuilderrc to figure out the basepath/basetgz,
architecture, distribution, and so forth and don't want git-pbuilder to
bypass all of that. Note that this turns off everything, including
--buildresult .., so if yoy use it, .pbuilderrc really has to
handle everything.
You can get the latest version from my scripts distribution page, and I suspect it will show up in the
git-buildpackage package before too long.
2011-12-25: Testing PAM modules
I maintain a couple of PAM modules, pam-afs-session and pam-krb5. I'm also
a big fan of test-driven development and automated regression testing, and
have been slowly converting all of my software over to using a test suite.
But therein lies a challenge, since PAM modules are hard to test.
First, Linux PAM doesn't offer a good way to run a module in a test
environment. Specifically, it doesn't offer a way to use an alternative
PAM configuration other than the system PAM configuration, so that one can
load only the module being tested (or it and some glue modules to set
things up). One has to resort to dedicated test systems that do things
like symlink part of their PAM configuration into a shared build tree.
Second, even if this problem were solved, Linux PAM doesn't (rightfully)
allow a lot of introspection and digging around in its guts. This makes
it harder to test whether data items, environment variables, and so forth
are set properly and make sense. It also doesn't provide facilities for
intercepting logging output that would otherwise go to syslog.
So, about a year and a half ago (and then with substantial improvements a
year ago), I wrote a fake PAM library to test pam-afs-session. This
library provides all of the PAM API that's called by a PAM module (at
least the modules I'm testing), but has a much-simplified interface and
allows test cases to look through its internals and be sure the module did
things properly.
This was great as far as it goes. But writing the wrapper code to invoke
a PAM module entry point and then check its return status, its logging,
and so forth was complicated, messy, and difficult to understand. It
also didn't handle testing prompting and other important aspects of PAM
modules. So earlier this fall I wrote the beginnings of a PAM test driver
that would move as much of the complexity as possible into shared code and
into configuration files, and since then I've been expanding and improving
it until it could handle the major testing required for pam-krb5 (which is
much more complex than pam-afs-session). Yesterday, I released the
results as part of rra-c-util.
Here's an example configuration file, this one for a pam-krb5 test that
simulates prompting the user for a password and then obtaining credentials
(but not creating a ticket cache):
[options]
auth = no_ccache
account = no_ccache
session = no_ccache
[run]
authenticate = PAM_SUCCESS
acct_mgmt = PAM_SUCCESS
open_session = PAM_SUCCESS
close_session = PAM_SUCCESS
[prompts]
echo_off = Password: |%p
[output]
INFO user %u authenticated as %u
That's a complete and fairly complex test. The test framework takes care
of running the listed entry points, passing them the given PAM
configuration, and checking the return status. It provides a conversation
function that checks the prompt and its type and replies with the text
after |. As you can see, it supports some escapes that are
replaced by data passed into the test suite driver, in this case the
password to use for testing. And it checks the logging output, including
the priority level, and similarly supports escapes there.
Running simple tests is very easy and is doable with a tiny C program.
Here's a complete test program from pam-krb5, one that tests functionality
that doesn't require a valid password and hence doesn't need to do much
setup:
#include <config.h>
#include <portable/system.h>
#include <tests/fakepam/script.h>
int
main(void)
{
struct script_config config;
plan_lazy();
memset(&config, 0, sizeof(config));
config.user = "root";
run_script_dir("data/scripts/basic", &config);
return 0;
}
That's all there is to it. That finds every configuration file like the
above in the tests/data/scripts/basic directory in the package
source and runs through each one, checking the results, and outputing the
results as TAP output (the protocol used for the Perl test suite, and also
my C TAP Harness package).
There are, of course, other things that it can do: more substitutions,
setting the authtok in advance of calling the module, and a callback so
that one can inspect PAM state before the PAM session is shut down. I've
started writing some
documentation, although so far it primarily covers the configuration file
syntax and the headers are the best reference for the API.
The source code of the pam-krb5 package
are the best reference so far. The library itself is part of
rra-c-util and, like the rest of that
package, is designed to be copied into the package that uses it and built
alongside it so that it can be linked easily with the test programs.
It's worth noting that there isn't a way to use it to test installed PAM
modules that are already bound to the system libpam library. The way one
builds test programs currently is to link the test program with the
individual objects making up the PAM module and then link with this
library and the TAP library, since that ensures the right symbols are
called. But it may be possible to do something tricky using LD_PRELOAD if
anyone feels inspired.
2011-12-26: filter-syslog 2.0
filter-syslog is the tool that we (two groups at Stanford) use to mail any
"interesting" lines from syslog to us each night. It runs as part of an
analyze rule in newsyslog.
Some time back in internal discussions, the idea came up that it would be
nice to filter out a range of lines, given a regex to match the start and
another to match the end. This would let us filter out blocks of lines
that are otherwise awkward to match, or whose specific details may change,
like system reboots. At the time, I thought this would be easy. It
turned out to require almost a complete rewrite of the state logic to deal
with several of the tricky cases (seeing the range start but not the range
end, several possible ranges that start with the same regex but have
different ends, a range inside another possible range that fails). But I
think I have all the cases taken care of now.
Separately, a co-worker had started using filter-syslog with raw regexes
to filter Apache error logs, something that I thought should be easier.
So I added support for parsing Apache error logs, stripping the timestamp
and client IP, and matching them as a "program" of apache-level
where level is the logging level.
These were big enough changes, particularly with the state rewrite, that
it felt like it warranted a 2.0 release.
You can get the latest version from the
filter-syslog distribution page.
I'll upload new Debian packages to my personal repository, but probably
not until tomorrow.
2011-12-29: rra-c-util 4.1
This release removes the Kerberos v4 portability code, which I've been
carrying around but unable to test for some time. No one is shipping
Kerberos v4 libraries any more, and hopefully few people are still using
it. I'm releasing my last (non-obsolete) package that still supported
Kerberos v4, and it seemed like a good time to drop it.
Also in this release is a replacement for a missing krb5_cc_get_full_name
function (in Heimdal for some time and MIT only recently), and a bug fix
in pam-util. The new pam-util option parsing code for handling Kerberos
times didn't work properly on platforms where a krb5_deltat was not a
long.
You can get the latest release from the rra-c-util distribution page.
2011-12-29: kstart 4.0
Between one thing and another, it had been almost two years since the last
kstart release, which surprised me when I finally started working on it
again. I'd been intending to put out a new release for some time, but
apparently never got around to it. As a result, there were a ton of
accumulated features and bug fixes, not to mention a huge update to the
testing and portability framework.
The main backward-incompatible change in this release is that I finally
dropped k4start from the distribution. I haven't been able to test it for
years, no one (or almost no one) is shipping Kerberos v4 libraries any
more, and I wanted to do a significant code restructuring. People who
really need it can use older releases.
k5start and krenew now both allow arbitrary ticket cache designators to be
passed with -k, rather than forcing the argument to -k to be a file cache,
and both canonicalize the ticket cache name (by asking the Kerberos
library for the real name) before passing it to subprocesses via the
environment. These features combined should allow them to work much
better with various non-file ticket caches.
k5start and krenew now both, when running as a daemon or when running a
command, default to staying running even if authentication fails. This
allows them to be more robust against temporary problems with contacting a
Kerberos KDC, and is similar to what krenew -i previously did. krenew
still exits by default if the ticket cache disappears or if the tickets
are no longer renewable; to make it stay running in those situations, use
krenew -i as before. Both k5start and krenew have a new -x option that
restores the previous behavior of exiting on any error.
k5start, when run with the -o, -g, or -m options, now writes out a
temporary ticket cache in the same directory, sets the ownership and
permissions, and then does an atomic rename, closing a possible race where
a process using that cache could temporarily not have access to it.
k5start and krenew both now propagate SIGINT to the child process when
running a command rather than just exiting. Signal handlers are now set
with sigaction, rather than signal, which will hopefully fix problems with
propagating multiple signals.
The embedded kafs library has been updated to the current rra-c-util
release, which adds support for Mac OS X and Solaris 11.
There are also a bunch of minor bug fixes and portability improvements,
particularly to the build system.
You can get the latest release from the kstart distribution page.
All posts