Posts for February 2012

2012-02-01: C++ symbols wrapup

I wanted to give a final update of my experiment with adding symbols files to C++ library packages.

In the end, I reverted the changes and have gone back to not providing a symbols file, and instead just using shlibs. Some of this is because of the general state of the available tools, and some of it is due to the nature of the packages I was working on. This set of packages has frequent SONAME bumps from upstream, so backward compatibility across Debian stable versions can't happen anyway. They also don't use symbol export control, which means that the number of exported symbols is quite large and contains a lot of things leaked from internal objects.

But there are also tool issues. The biggest that I ran into is that symbols appear and disappear in the export list with different versions of the compiler, and the pkg-kde-tools utilities, while excellent, still interpret those changes as architecture-specific changes. So the symbols file accumulates supposedly arch-specific variations, which are actually about what version of the compiler happened to be installed on that buildd when the package was built. This also means that if a package that has had its symbols file updated for one architecture is built again on the same architecture but on another buildd with a different g++, it will often FTBFS since a different set of symbols will be present.

Obviously, this is too fragile to be maintainable.

Further debian-devel discussion pointed out that these symbol variations are probably all inline functions, where the compiler may or may not output a weak version of the symbol if it chooses not to inline it in some particular case. The right thing to do for these libraries (not all, since there are other uses of weak symbols) is to mark all weak symbols as optional and not care if they randomly appear and disappear.

There are also some 32-bit vs. 64-bit variations that are not captured by the subst mechanism, which resulted in the symbols file developing a list of architecture restrictions that amounted to "all 32-bit systems" or "all 64-bit systems." These explicit lists are also fragile, and will break with each new architecture introduced in Debian.

I filed three bugs with pkg-kde-tools with suggestions for improving the tools for other people going forward:

#657805 — pkgkde-getbuildlogs gets truncated logs
#657806 — representation of covariant return thunks
#658333 — option to mark weak symbols as optional

but I think that the level of work and remaining fragility doesn't make sense for a lot of C++ libraries, at least right now without more direct support in dpkg-gensymbols and other tool improvements. I'm therefore also planning on changing the proposed Policy patch for symbols to make it more clear that shlibs is an acceptable alternative for C++, and to patch Lintian to suppress the info tag for no symbols file for C++ libraries.

2012-02-19: Debian, licenses, and license-count

A few people have been using the license-count tool that I wrote to analyze the breakdown of licenses in Debian and draw some conclusions. Unfortunately, some of those conclusions can't be drawn from the output of this specific tool (which doesn't make them wrong, just unproven). So here are a few cautions.

license-count was written for a specific purpose: to see how many references there are to a particular license in Debian so that we had some facts behind our discussion of whether to include it in common-licenses. It was not written to classify packages into particular categories, only to find out how many copies of a license we could save by adding it to common-licenses. This means that using it to determine how much software in Debian is covered by a particular license fails in the following ways:

  1. license-count numbers cannot be added. Since the goal is to find every reference to a given license, each package is not classified as under one and only one license. If a package has material under the GPL, the LGPL, and the GFDL, it will add one to each of those counts. This means that you cannot determine, from its output, questions like "how many packages are covered under one of the GPL family of licenses?" You will double- or treble-count packages when you add the numbers together.

  2. license-count counts any reference, not just "important" ones. If the license is referenced in debian/copyright, that counts, since the goal is to see how many packages might benefit from being able to reference it in common-licenses. This may not be the license that the package is under, just some small portion of the package.

    For example, all of my packages that use Autotools add to the GPL count because I include all the source code licensing statements, including the statements for libtool and the Automake helper scripts, which say that the files can be released under either the GPL or the license of the including package. These packages are generally under a BSD-style license (or in some cases the Apache 2.0 license), not under the GPL, but since the reproduced license text has a reference to the GPL, license-count adds one to the GPL column for each of those binary packages. For most purposes for counting GPL-covered software, these are false positives. I don't know how many people are as complete in documenting license statements as I am, but just the binary packages that I maintain probably count for at least 50 false positives because of this alone.

  3. license-count numbers include dual-licensing. Whether this matters for your analysis or not depends on what question you're asking, of course. If you're asking whether the package is covered under a copyleft license, including the thousands of packages that are dual-licensed under the Artistic license and the GPL may be okay, since the Artistic license is a sort of copyleft (although it allows you to avoid the copyleft requirement by renaming executables, providing the originals, and documenting your changes). But, for example, one of my packages is dual-licensed under a BSD-style license and the GPL (for reasons that aren't worth getting into). This package is not covered by copyleft under any reasonable definition of the term, but license-count will count it as a package using the GPL.

In order to draw interesting conclusions like "how much software in Debian is protected by copyleft," I'm afraid that you'd have to write a much more sophisticated tool than license-count.

2012-02-20: Debian packaging updates

Now is the point in the Debian release cycle where I usually try to go through my packages and bring them generally up to date. In case other people are doing the same thing, here are a few new facilities or techniques that I'm rolling out across my packages. (This is apart from the obvious stuff, like multiarch where appropriate and debhelper 9, and the older stuff, like using dh.)

It's also worth mentioning that Ubuntu was responsible for breaking a lot of ground here. Due to bug reports and patches submitted from Ubuntu, several of my packages already had hardening build flags and --as-needed issues fixed before this round of packaging refresh, which made adding these features much easier than it would be otherwise.

2012-02-22: Debian Policy 3.9.3.0

I've just uploaded version 3.9.3.0 of Debian Policy. The really big change in this version is the 1.0 version of the copyright format specification, but that will get a separate announcement once the www team has had a chance to put it in place on the web site.

There are a bunch of other fixes, though, including some normative changes. Here's the upgrading checklist (duplicated from my announcement, but formatted a bit prettier):

2.4

New archive sections education, introspection, and metapackages added.

5.6.8

The Architecture field in *.dsc files may now contain the value any all for source packages building both architecture-independent and architecture-dependent packages.

7.1

If a dependency is restricted to particular architectures, the list of architectures must be non-empty.

9.1.1

/run is allowed as an exception to the FHS and replaces /var/run. /run/lock replaces /var/lock. The FHS requirements for the older directories apply to these directories as well. Backward compatibility links will be maintained and packages need not switch to referencing /run directly yet. Files in /run should be stored in a temporary file system.

9.1.4

New section spelling out the requirements for packages that use files in /run, /var/run, or /var/lock. This generalizes information previously only in 9.3.2.

9.5

Cron job file names must not contain "." or "+" or they will be ignored by cron. They should replace those characters with "_". If a package provides multiple cron job files in the same directory, they should each start with the package name (possibly modified as above), "-", and then some suitable suffix.

9.10

Packages using doc-base do not need to call install-docs anymore.

10.7.4

Packages that declare the same conffile may see left-over configuration files from each other even if they conflict.

11.8

The Policy rules around Motif libraries were just a special case of normal rules for non-free dependencies and were largely obsolete, so they have been removed.

12.5

debian/copyright is no longer required to list the Debian maintainers involved in the creation of the package (although note that the requirement to list copyright information is unchanged).

copyright-format

Version 1.0 of the "Machine-readable debian/copyright file" specification is included.

mime

This separate document has been retired and and its (short) contents merged into Policy section 9.7. There are no changes to the requirements.

perl

Packages may declare an interest in the <perl-major-upgrade> trigger to be notified of major upgrades of perl.

virtual

ttf-japanese-{mincho,gothic} have been renamed to fonts-japanese-{mincho,gothic}.

Please review and update your packages as necessary the next time you make an upload.

2012-02-24: Copyright format 1.0 published

Four and a half years after Sam Hocevar started a draft proposal on the Debian Wiki, we've published revision 1.0 of the machine-readable format for debian/copyright files. The format (as DEP-5) has been in widespread use for some time, but there have been multiple versions of the format and multiple transitional versioned URLs. Everyone can now update to the same version of the document and use a stable format URL.

Below is the text of the annoucement that I just sent to debian-devel-announce.

Version 1.0 of the machine-readable format for debian/copyright files (the culmination of the DEP-5 process) has now been published. The canonical URL (and also the URL to use in the Format field in such files) is:

http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/

For those who have adopted various iterations of the DEP-5 format for your packages, please consider updating to this format the next time you revise your package.

Use of this format in Debian packages is optional.

Thanks to all of the people who have contributed to this work over its long evolution. Particular thanks go to Steve Langasek, Charles Plessy, and Lars Wirzenius, who kept this document moving through its extended DEP discussion process, and Sam Hocevar, who started this work in 2007.

The copyright format specification is now being maintained as part of the debian-policy package and will follow the same update process that's used for other parts of Policy. See:

https://wiki.debian.org/PolicyChangesProcess

for that process.

For this specification, we're trying something new for Debian technical policies. This document is versioned, and older versions of the specification will be maintained to not invalidate older references in copyright files that have not been moved to newer versions. This is similar to the way that specifications are handled by some other projects, particularly the freedesktop.org standards.

The plan, going forward, is that informative changes (changes that do not change the requirements or the contents of compliant copyright files) will be made following the same relaxed procedure as other informative changes to Policy, and new versions will only informative changes will be published with the same version number. So the existing 1.0 document may receive further improvements in wording, examples, or similar changes that don't modify its meaning.

Normative changes (changes that change requirements for compliant files, which includes addition of new standardized license short names) will follow the normative Policy change process, including formal seconds.

Once normative changes are accepted, the next release of this specification will receive a new version number (1.1, for example). The previous version, prior to the normative changes, will be frozen, and from that point forward will not be changed except as required to continue to publish it. This includes further informative changes. We will continue publishing these frozen older versions of the standard on www.debian.org and in the debian-policy package for some time: possibly forever, and at least as long as they're still referred to by packages distributed by the project.

The hope is that this will provide a good balance between specification stability, editing for clarity, and specification changes to meet changing requirements, while not invalidating the URLs in older files and while continuing to provide the information required to interpret older files.

2012-02-29: C TAP Harness 1.10

The only change in this release is to add a new bstrndup function to the C TAP library. Like the other b* memory allocation functions, this emulates the (semi-standard) strndup function but calls sysbail on memory allocation failure. It reimplements strndup to not assume that the system C library has it (since it's missing on Mac OS X).

You can get the latest version from the C TAP Harness distribution page.

2012-02-29: rra-c-util 4.2

The primary changes in this release are in the networking utility library and in the various provided TAP add-ons for testing software that uses Kerberos and remctl.

In the networking library, new network_read and network_write functions have been added to read and write from network sockets with an optional timeout. Various problems with the network utility test suite have also been fixed.

The Kerberos and remctl TAP add-ons have been significantly refactored and improved. The kerberos_setup function now returns a struct of configuration and supports being told what configuration is required. skip_all will be called automatically if required configuration is missing. The native Kerberos and kinit versions of the test suite setup have also been merged and the correct alternative is chosen at compile time. The remctl TAP add-on has been updated to match, now gets the path to remctld from a #define so that each caller doesn't have to provide it, uses the new test file functions to find its configuration, and supports running remctld under fakeroot.

This release also has a bug fix for the replacement strndup (it previously assumed the source string was nul-terminated) and some improvements to the xstrndup checked version.

You can get the latest release from the rra-c-util distribution page.

2012-02-29: remctl 3.1

The primary change in this release is to add a new remctl_set_timeout function to the remctl client library and similar functions to the Perl, PHP, Python, and Ruby bindings. When set, the timeout affects all subsequent network operations, including connect.

The remctld server, thanks to Andrew Mortensen, now supports a new configuration option that sets the user as which to run the command. When set, remctld will change to that user (including their groups) before running the command, freeing command authors from having to write the the code to change users.

Thanks to the code for timeout handling, the timeout on the remctld server is now an hour timeout between messages from the client rather than an hour limit on the entire connection.

There are also some minor fixes to some of the internals of the PHP and Python bindings.

You can get the latest release from the remctl distribution page.

Last modified and spun 2017-08-01