C TAP Harness 4.3

This is my test harness and C library for supporting TAP (Test Anything Protocol, the same used by Perl) tests for C projects.

The big change in this release is test options: a test list passed to runtests can now include space-separated options after the name of the test to run. The two supported options are valgrind, which controls whether the test can be run under valgrind, and libtool, which further controls how the test is run under valgrind.

The background on this is a bit interesting.

I've had Makefile targets to run test suites under valgrind for various packages for a while, using valgrind's --trace-children=yes option and running runtests under valgrind. However, this has a serious problem: valgrind tries to trace into every child, including shell scripts or Perl scripts used in the test suite. This results in valgrind logs that are basically always unuseful: lots of false positives or uninteresting small memory leaks in standard system tools.

For projects that use Libtool, the situation becomes far worse, since Libtool wraps all the executables in the build tree with shell scripts that set up an appropriate library path. valgrind traces into those shell scripts and all the various programs they run. As a result, running valgrind with --trace-children=yes on a non-trivial test suite such as remctl's could easily result in 10,000 or more valgrind log files, most of which were just noise.

As a result, while the valgrind target was there, I rarely ran it or tried to dig through the results. And that, in turn, led to security vulnerabilities like the recent one in remctl.

Test options are a more comprehensive fix. When the valgrind test option is set, C TAP Harness will look at the C_TAP_VALGRIND environment variable. If it is set, that test will be run, directly by the harness, using the command given in that environment variable. A typical setting might be:

VALGRIND_COMMAND = $(PATH_VALGRIND) --leak-check=full             \
        --log-file=$(abs_top_builddir)/tests/tmp/valgrind/log.%p  \
        --suppressions=$(abs_top_srcdir)/tests/data/valgrind.supp

and then a Makefile target like:

# Used by maintainers to run the main test suite under valgrind.
check-valgrind: $(check_PROGRAMS)
        rm -rf $(abs_top_builddir)/tests/tmp
        mkdir $(abs_top_builddir)/tests/tmp
        mkdir $(abs_top_builddir)/tests/tmp/valgrind
        C_TAP_VALGRIND="$(VALGRIND_COMMAND)"                    \
            C_TAP_LIBTOOL="$(top_builddir)/libtool"             \
            tests/runtests -l '$(abs_top_srcdir)/tests/TESTS'

(this is take from the current version of remctl). Note that --trace-children=yes can be omitted, which avoids recursing into any small helper programs that tests run, and only tests with the valgrind option set will be run under valgrind (the rest will be run as normal). This cuts down massively on the noise. Tests that explicitly want to invoke valgrind on some other program can use the presence of C_TAP_VALGRIND in the environment to decide whether to do that and what command to use.

The C_TAP_LIBTOOL setting above is the other half of the fix. Packages that use Libtool may have tests that are Libtool-generated shell wrappers, so just using the above would still run valgrind on a shell script. But if the libtool test option is also set, C TAP Harness now knows to invoke the test with libtool --mode=execute and then the valgrind command, which causes Libtool to expand all of the shell wrappers to actual binaries first and then run valgrind only on the actual binary. The path to libtool is taken from the C_TAP_LIBTOOL environment variable.

The final piece is an additional test that scans the generated valgrind log files. That piece is part of rra-c-util, so I'll talk about it with that release.

There are two more changes in this release. First, borrowing an idea from Rust, diagnostics from test failures are now reported as the left value and the right value, instead of the seen and wanted values. This allows one to write tests without worrying about the order of arguments to the is_* functions, which turned out to be hard to remember and mostly a waste of energy to keep consistent. And second, I fixed an old bug with NULL handling in is_string that meant a NULL might compare as equal to a literal string "(null)". This test is now done properly.

The relationship between C TAP Harness and rra-c-util got a bit fuzzier in this release since I wanted to import more standard tests from rra-c-util and ended up importing the Perl test framework. This increases the amount of stuff that's in the C TAP Harness distribution that's canonically maintained in rra-c-util. I considered moving the point of canonical maintenance to C TAP Harness, but it felt rather out of scope for this package, so decided to live with it. Maybe I should just merge the two packages, but I know a lot of people who use pieces of C TAP Harness but don't care about rra-c-util, so it seems worthwhile to keep them separate.

I did replace the tests/docs/pod-t and tests/docs/pod-spelling-t tests in C TAP Harness with the rra-c-util versions in this release since I imported the Perl test framework anyway. I think most users of the old C TAP Harness versions will be able to use the rra-c-util versions without too much difficulty, although it does require importing the Perl modules from rra-c-util into a tests/tap/perl directory.

You can get the latest release from the C TAP Harness distribution page.