< Debian Best Practices | Russ Allbery > Technical Notes > Debian | Debian Packaging Tools > |
Introduction
Repository Layout
Configuration
Distribution Upgrades
Remote API
Audits
At Stanford, we used Debian heavily for nearly all of our central servers, for services ranging from authentication through web and file services to end-user computing and application-specific middleware. We also tried to use Debian-native tools as much as possible, including using Debian's packaged version of software rather than building our own and creating our own Debian packages where something is missing. We had a rich array of packages for locally-developed software.
We also had an internal requirement that any executable be installed via a Debian package. This is not strictly necessary — we could have installed most scripts via Puppet instead — but the requirement ensures that everyone is familiar with Debian packaging, allows better handling of man pages, and allows those packages to become more complex when needed.
We therefore needed an internal Debian repository to manage our packages. This page documents the repository setup I developed for Stanford, and some of the issues that we ran into, in the hope that the information will be useful to others.
We used two separate internal repositories: debian-stanford and debian-local.
We considered using a single repository, but we occasionally had software with licensing restrictions that can only be installed by central servers or that cannot be legally redistributed. We also had a lot of internal packages for our various servers that are neither useful nor supported for the campus as a whole. That led us to separating the packages into two repositories.
In debian-stanford, we put packages potentially of use to the campus as a whole, such as locally-developed tools or metapackages that other Debian or Ubuntu users around campus may use. These are only the packages that we considered supported and would answer help tickets about.
In debian-local, we put all the other packages: packages of software we used on our servers that isn't in Debian for some reason, packages of locally-developed software only used internally, and the packages that hold scripts and utilities for use on our servers. Access to this repository was restricted to our server IP space (more or less).
We occasionally did local backports of software and uploaded them to debian-local when we needed to use a backported version, but since I had upload rights to backports.debian.org, I more frequently just uploaded the backport there and then used pinning to pull the packages from there. At sites where there isn't a Debian Developer on staff, I would expect more use of the local repository to hold local backports.
In both repositories, we followed a distribution layout matching that of the Debian archive: unstable, testing, stable, and oldstable, identified by codename, with symlinks that change for each release. We pointed all of our servers to the repositories by codename so that, when a new version was released, they didn't automatically pick up the new packages. We automatically copied into testing every package uploaded to unstable rather than waiting some period of time like Debian does, since usually we had very few systems running unstable or testing. We generally uploaded packages to unstable and then pulled them or backported them into stable, although sometimes if we were lazy we'd upload directly to stable. We tried to use unstable first, though, for one of the same reasons that Debian does: it means the upgrade path when the next stable releases is taken care of.
In the debian-local repository, we also created some separate distributions for particular servers. For various reasons, we had to use our own packages for OpenLDAP on our directory servers and for Postfix on our mail routers, but we didn't want to use our customized packages on all of our servers for things like the regular OpenLDAP libraries. We therefore had distributions named stable-email and stable-ldap (actually, named by the codenames with symlinks for those names) and uploaded our customized packages only to those repositories.
Although we had some Ubuntu systems, we didn't maintain packages for specific Ubuntu distributions. Instead, we pointed our Ubuntu systems at the "closest" Debian distribution for that Ubuntu release. That mostly worked okay.
Both of our repositories were managed by reprepro, with which we were extremely happy. They were completely separate from each other, including configuration, temporary directories, and the entire rest of the structure. They did, however, share a repository signing key.
A typical stanza in our distributions
configuration file looked
like this:
Codename: squeeze Suite: stable Origin: Stanford Label: Debian Description: Stanford supplemental Debian repository Architectures: amd64 i386 source Components: main contrib non-free Pull: auto-stable SignWith: FDF37CD4279D4962 DebIndices: Packages Release . .gz .bz2 DscIndices: Sources Release .gz .bz2 Contents: .gz .bz2 Tracking: all Log: --changes /usr/bin/reprepro-notify
(Obviously this is from when squeeze was stable.) This is mostly obvious.
The Pull setting will be discussed in a moment. We enableed source
package tracking and have been very happy with it.
/usr/bin/reprepro-notify
is a very simple shell script that mails
the *.changes
file for a new upload to an internal reporting
address.
We used the following pulls
configuration file:
Name: auto-stable From: sid FilterList: hold pull-stable Name: auto-oldstable From: sid FilterList: hold pull-oldstable Name: everything From: sid
The everything
pull was used for testing. The others were for
stable and oldstable, obviously, and reference files that define which
packages to pull. For internal packages that bundled together scripts for
a particular service, we found it easiest to automatically pull that
package into stable whenever it was uploaded.
Finally, our incoming
configuration:
Name: default IncomingDir: /srv/repos/stanford/incoming TempDir: /srv/repos/stanford/tmp Allow: oldstable>lenny stable>squeeze unstable>sid Multiple: yes
This is for debian-stanford; the version for debian-local was similar but had more Allow mappings for our service-specific repositories.
When a new stable version was released, we did basically the same thing that Debian does:
Update the distributions file to no longer pull everything into the previously testing, now stable distribution and switch it over to the auto-stable rule.
Change the now-oldstable distribution over to the auto-oldstable rule.
Change the suite configuration in the distributions file to move the stable and oldstable names.
Create a new distribution for the new testing, configure it with the everything pull rule, and then do a reprepro pull to populate it from unstable.
It's important to point all systems at the distribution codename rather than directly at stable or oldstable, since otherwise the available packages can change unexpectedly without any configuration changes after the release of a new version of Debian.
All of the staff making uploads also had root access on the repository server and could use reprepro commands directly, but that requires configuring the location of the GnuPG keyring used to sign the repository metadata and selecting the right repository to work with. It also requires logging on to the system, and we tried to avoid logging on to servers wherever possible in favor of exposing useful APIs via remctl. Finally, many of the reprepro commands should only be used by experts or should not normally be used in our repository.
We therefore wrote a wrapper around reprepro that adds an additional argument before each command to specify which repository to act on, can be run via remctl, and which sets up the location of the GnuPG keyring. That wrapper initially only exposed the commands copy, copysrc, list, ls, remove, and removesrc currently, and only added new commands when we had regular need of them.
The wrapper also added a new command, clean, which removed any files left behind in the incoming directory, so that people can remotely clean up after failed uploads that were rejected for some reason.
The wrapper isn't currently available outside of Stanford, but will be in the future (as soon as I have time to clean it up a bit and adapt it for my personal repository).
We also had a separate script that ran processincoming
and
pull
after a new upload. Eventually, this will be rolled into the
same wrapper.
When I left Stanford, I'd only done some preliminary work on audits to check repositories for consistency and sanity. At the time I left, there were only three audits, all of which were implemented by our wrapper and which were built on top of the reprepro commands plus some other standard Debian tools like rmadison:
Shows the list of packages that aren't built for all supported architectures. This just runs reprepro build-needing, except that it does so for each codename and each architecture as one command.
Lists packages that are present in both debian-local and debian-stanford. Generally this is a mistake caused by someone being too used to uploading to local and forgetting to upload to stanford something that is used by campus.
Lists packages whose version is older than the package in the same Debian distribution, as determined by rmadison. We use this to locate cases where we've locally packaged something that's now in Debian or backported something for which a newer version is now available in the stable distribution.
< Debian Best Practices | Russ Allbery > Technical Notes > Debian | Debian Packaging Tools > |