| < Debian Best Practices | Russ Allbery > Technical Notes > Debian | Using Git for Debian > |
Introduction
Packaging and Upstream Together
Subversion
svk with Subversion
Uploading and Local Repositories
Special Package Build Systems
Packaging Individual Scripts
Other Useful Tools
One of the features of Debian packaging over other Linux distributions is its extensive and excellent standards documentation (independent of method) and a rich set of tools that can produce packages that meet those standards. This abundance of riches can be rather staggering and confusing, though, and I keep discovering new tools that I wish I had known about months ago. This document is therefore my attempt to catalog the various useful tools and approaches to Debian packaging I've used, as well as some suggestions and recommendations for approaches I've found to work well.
I am also the upstream maintainer of many of the software packages that I package for Debian, so I'll also discuss ways of maintaining the Debian packaging in the same repository with the software package itself. This poses a few interesting challenges.
In the following, I assume a basic knowledge of packaging terminology, and much of this will likely only be useful to someone who has already done a few basic packages and is familiar with the overall process. You should probably have read the New Maintainer's Guide, Debian Policy Manual, and the Debian Developer's Reference before reading this, or at least be familiar with the contents of those documents.
When I'm also the upstream maintainer of a software package that I'm packaging for Debian, I find it most convenient to keep the packaging files in the same revision control system as the rest of the package. However, there are a few issues to be aware of:
The debian directory and the packaging files should not be included in the upstream release of the package. There are several reasons for this: you will often do multiple Debian package releases for the same upstream source to fix packaging-only problems but won't want to do a new source release each time you change something, having outdated Debian packaging rules in the source distribution can be confusing to users, and someone who wants to build the package for Debian is far better off retrieving the Debian source package than using the files in your regular release.
I use a few different methods for generating release tarballs (a topic
for another document), but the simplest one I've found for small
packages is to use rsync to copy the source into a release directory.
With that method, just pass --exclude /debian/ to rsync when
generating the release.
The .orig.tar.gz file in the Debian source package should be identical to the upstream release tarball. In fact, you want to use a copy of your last upstream release tarball to go with the Debian package rather than even trying to generate a new one, since it's too hard to get a newly generated tarball to match. That way, any signatures, MD5 checksums, and the like can still be verified.
What I do to build a new Debian package is pull the last released tarball, untar it, and then export just the debian directory from the package revision control system to form the build tree. This works well and is easy to automate. The one drawback is that all changes specific to Debian have to be localized in the debian directory. Usually that's fine, and any changes necessary to the rest of the code can be a reason for a new release of the package. Sometimes, though, a change specific to Debian is necessary and can't be included in the regular release. In that case, use a patch management system that keeps all the files in the debian directory except during the build.
You can still use systems like svn-buildpackage to build Debian packages when using a combined repository, but it's a bit more awkward and svn-buildpackage won't do as much for you. You don't want to use svn-inject or svn-upgrade, only the svn-buildpackage component, and you'll have to handle updating the tarballs directory yourself.
If this all sounds like too much hassle, consider maintaining the Debian packaging completely independent of the upstream package. This means having two separate repositories, shuffling patches around, and keeping multiple copies of the source tree. However, it also means that you can use all of the same techniques that you use for any other package.
If you're feeling experimental or cutting-edge, you may want to try one of the new distributed revision control systems for maintaining the Debian package. I'm too familiar with CVS and TLA hurt my brain when I gave it a try, so I mostly stick with Subversion. It works the way that I expect it to, even if the merging isn't quite as good.
The basic and easiest way of maintaining Debian packages in Subversion is to use svn-buildpackage. Start by creating a Subversion repository for Debian packaging, create the initial package (either from scratch or by downloading the existing package with apt-source if you're adopting a new one), and then put the package files in the directory you're planning on using as your working area for packaging. Run svn-inject on the .dsc file, passing it the URL of your repository as the second argument, and it will set up the initial repository structure and tag the current Debian package. It will also create a build-area and tarballs directory in the current directory and give you an initial checkout. The upstream orig tarball will be copied into tarballs. You can delete the Debian package now.
When making changes to the Debian package, I generally change the debian/changelog file at the same time and then run debcommit from the top level of the working directory. It commits all modified files and uses the changelog entry as the commit message.
To do the package builds, it's best to use pbuilder to make sure that the
package builds in an isolated environment. I use the following as my
~/.svn-buildpackage.conf file:
svn-builder=pdebuild --buildresult /home/eagle/dvl/debian/build-area
svn-lintian
svn-postbuild=rm /home/eagle/dvl/debian/build-area/*_source.changes
The last line cleans up the extra changes file created by pdebuild so that you can run debsign and dput on *.changes in the build-area directory. You'll want to change the path given to pdebuild to match your working directory.
The drawback to svn-buildpackage is that it doesn't have great merging support for new upstream versions. Subversion's default merge support isn't particularly great. svk is a wrapper around Subversion (and other revision control systems) that supports much better merging and also adds distributed development.
You can use svk with the Subversion repository created by svn-buildpackage, but that has the drawback that you have to keep the orig tarball around outside of the revision control system and you can't really use svn-upgrade as easily because you want to instead use svk to do the merge. Normally, svn-buildpackage creates a package structure like:
branches/upstream/
tags/
trunk/
Instead, create a repository structure like:
branches/
tags/
trunk/
vendor/current/
and then import the original source into vendor/current/package. In other words, don't put the source directly into vendor/current; instead, create another level of subdirectory matching the name of the package and import the source into that. I generally then svk cp the vendor/current directory to vendor/version to tag that import of the upstream source.
Then, use svk smerge to merge vendor/current to trunk. This will leave you with trunk/package containing the package source. Check out trunk, and then put the orig tarball (with the proper Debian name) in the top level directory of your checkout (at the same level as the package directory containing the package source). Remember to mark the orig tarball as a binary with:
svk propset svn:mime-type application/octet-stream *.orig.tar.gz
Now, you can do development in your checkout of the trunk and commit changes to the upstream source, relying on svk's smerge to handle merges with new versions. And, because of the extra nesting in the directory structure, you can then use pdebuild or debuild directly to build the package. Better yet, anyone else using this repository can just check out a working copy and have everything, including the orig tarball, needed to build the package.
When a Debian package is finished and uploaded, svk cp trunk to tags/version. When a new upstream version is released, follow exactly the same procedure as given above for starting off the repository. Branches can be used to maintain stable security updates, long-lived experimental branches, or similar forks in development.
One other nice feature of svk over svn-buildpackage besides distributed
development and better merging is that svk gets rid of all the .svn
metadirectories. svk keeps that metadata elsewhere, so grep -r and
similar commands work as expected. (The tradeoff is that you can't just
move checkouts around; svk relies on knowing where they are.)
A lot of the packaging work I do is specific to Stanford, packaging local software or Stanford variants of packages that we customize. I also package some of my own software that isn't of broad enough interest to warrant uploading to Debian, and it's sometimes nice to have a staging area where I can point apt at a repository before I actually upload to Debian. (This is particularly true before one is a Debian developer and one needs someplace to put the package for a sponsor to download it to build, sign, and upload.)
There are a bunch of packages that maintain a local repository, and I tried several, but I had trouble getting many of them to work. I finally settled on debarchiver and have contributed some improvements to it so that it now does most everything one would want, although it doesn't support package pools (it still uses the old package layout). But apt copes with that without any trouble, and the only drawback is some additional disk space if you have the same package in both a stable and unstable repository.
I use dput to do the package upload, with a configuration like:
[DEFAULT]
method = local
hash = md5
allow_unsigned_uploads = 0
run_lintian = 0
run_dinstall = 0
check_version = 0
[local]
method = local
fqdn = localhost
incoming = incoming
post_upload_command = /usr/bin/debarchiver -x -i incoming
in ~/.dput.cf, where incoming is the path to a
temporary directory to put packages for debarchiver to pick up. I don't
run lintian at upload time because I run it at build time. I put the
following configuration in ~/.debarchiver.conf:
# Don't install the changes files.
$cinstall = '';
# Install by default into the staging area for my apt repository.
$destdir = '/afs/ir/users/r/r/rra/public/debian/dists';
# Point apt-ftparchive at its cache files.
$cachedir = '/home/eagle/data/debian/cache';
# Send me mail whenever something is successfully installed, for no good
# reason other than I think the mail is neat.
@mailtos = qw/rra@stanford.edu/;
# Sign the archive with my public key.
$gpgkey = 'rra@stanford.edu';
# Additional information for generating Release files.
$release{origin} = 'rra';
$release{label} = 'Debian';
$release{description} = "Russ Allbery's personal Debian package archive";
You'll want to change the destdir (the actual repository location) and the path to the cache directory. The repository location should be something web-accessible, of course, so that other people can point apt at it. You'll also want to change the origin and description to something specific to your repository and change the public key to your public key.
This configuration requires that all packages are signed (using debsign on the changes file) before upload, and also signs the repository files so that the new apt signed repository support will work. The only drawback is that this requires typing your private key passphrase three times for each package build and upload.
debarchiver by default creates separate repositories for the current unstable, testing, and stable (with the appropriate nicknames). I usually symlink testing to unstable for my personal repository; debarchiver deals correctly with this, and it's usually close enough. Since debarchiver doesn't have any way of migrating packages from one repository to another easily, this is the simplest way of handling it, even though if your packages are always built against unstable, they sometimes won't be installable on testing systems. When a new stable is released, I just move the stable link to the old unstable and then create a new unstable.
For most packages, making necessary modifications directly to the upstream source and then using either svn-buildpackage and the default Subversion merge or using svk (or some other more advanced revision control system) to handle more complex merges is sufficient. However, this results in a monolithic diff file and nothing else for someone else who is trying to make changes to the package (for an NMU, perhaps), and means that you have to take apart that large diff file to submit patches upstream. The utilities in the patchutils package can help with that process, but even those utilities don't help much if there are overlapping patches to the same file that are conceptually separate.
This is where more complex Debian build systems come in. I've used two: dbs and dpatch. Both have a similar basic concept, namely to not include any modifications to the upstream source in the Debian package diff and instead ship patches in a subdirectory of the debian directory and apply them at build time.
dbs is the most comprehensive in its changes to how the package is maintained. When using dbs, the orig tarball isn't a copy of the upstream tarball, but rather is a tarball containing the upstream tarball. dbs then unpacks that tarball as part of the build process. There are variants on this approach (cdbs being the most notable) and some people like it, but I find it complicated and don't like how it makes it more difficult to just start hacking on the package when testing or trying something.
My preferred solution is dpatch. When using dpatch, everything else about the package looks about the same, but you include some boilerplate in debian/rules and create a debian/patches directory to hold the upstream modifications as patches. The dpatch-edit-patch command can then be used to modify or create new patches.
The substantial advantage of one of these approaches is that the patches are all separated out to send upstream when appropriate; the disadvantage is that these patches have to be ported (and ideally unfuzzed) for new upstream releases, and there's less automation available to do that work.
Usually the easiest way to handle individual scripts is to just put them
in /usr/local/bin or /usr/local/sbin. However, sometimes
it's nice to version them, to easily upgrade them, to easily handle the
man pages, and to otherwise get all the benefits of the Debian package.
When packaging a script, I use a debian/rules file with a get-orig-source target something like this:
URL = http://archives.eyrie.org/software/cvs/svnlog
VERSION = `awk '/Id: / { print $$6; exit }' svnlog`
get-orig-source:
wget -O svnlog $(URL)
chmod 755 svnlog
mkdir svnlog-$(VERSION)
cp -p svnlog svnlog-$(VERSION)/
tar cfz svnlog_$(VERSION).orig.tar.gz svnlog-$(VERSION)
rm -r svnlog-$(VERSION)
This assumes that the upstream script uses a CVS Id string for versioning. I do for all of my scripts. If not, you'll have to track the version some other way (perhaps by modifying debian/rules to add the latest version each time a new version is released). You can then run this target to grab the latest version and generate a useful orig tarball. I then commit the new script and use svn-buildpackage to build the package.
If the script uses POD for documentation, you can also generate a man page using something like:
build: build-stamp
build-stamp:
dh_testdir
pod2man --release=$(VERSION) --center="User Commands" \
svnlog > svnlog.1
touch build-stamp
in debian/rules, with the corresponding install rules to install the man pages (perhaps using dh_installman).
The most useful tools I've found other than the basic infrastructure mentioned above are the utilities that come in the patchutils package. These utilities manipulate diff files in various ways. Probably the most useful is filterdiff, which allows one to extract or suppress sections of a diff. This is extremely useful for taking apart a Debian package diff, or removing all the debian directory modifications to find the changes to the upstream source. There are also utilities to remove fuzz on diffs, split diffs apart, and similar operations.
devscripts is the other obvious collection of utilities. I mentioned debcommit above as a good way of committing changes to Debian packages with an appropriate change message. debuild is the easiest way of doing a quick package build, although I always use pbuilder and its pdebuild command for doing real builds. uscan uses a debian/watch file to check for a new upstream version (and including those watch files also makes your Debian QA summary page more useful). bts is a great command-line interface to the Debian bug-tracking system. wnpp-alert lets you know about orphaned packages that you use.
Since I use XEmacs to do all my development, I also use the dpkg-dev-el package, which sets up major modes for the various Debian control files and does some simple syntax highlighting. It's the most useful for editing the changelog file, but the syntax highlighting makes the control, copyright, and README.Debian files look pretty, and it updates the date at the bottom of the latter.
| < Debian Best Practices | Russ Allbery > Technical Notes > Debian | Using Git for Debian > |