Debian Private Repositories

Table of Contents

Introduction
Repository Layout
Configuration
Distribution Upgrades
Remote API
Audits

Introduction

At Stanford, we used Debian heavily for nearly all of our central servers, for services ranging from authentication through web and file services to end-user computing and application-specific middleware. We also tried to use Debian-native tools as much as possible, including using Debian's packaged version of software rather than building our own and creating our own Debian packages where something is missing. We had a rich array of packages for locally-developed software.

We also had an internal requirement that any executable be installed via a Debian package. This is not strictly necessary — we could have installed most scripts via Puppet instead — but the requirement ensures that everyone is familiar with Debian packaging, allows better handling of man pages, and allows those packages to become more complex when needed.

We therefore needed an internal Debian repository to manage our packages. This page documents the repository setup I developed for Stanford, and some of the issues that we ran into, in the hope that the information will be useful to others.

Repository Layout

We used two separate internal repositories: debian-stanford and debian-local.

We considered using a single repository, but we occasionally had software with licensing restrictions that can only be installed by central servers or that cannot be legally redistributed. We also had a lot of internal packages for our various servers that are neither useful nor supported for the campus as a whole. That led us to separating the packages into two repositories.

In debian-stanford, we put packages potentially of use to the campus as a whole, such as locally-developed tools or metapackages that other Debian or Ubuntu users around campus may use. These are only the packages that we considered supported and would answer help tickets about.

In debian-local, we put all the other packages: packages of software we used on our servers that isn't in Debian for some reason, packages of locally-developed software only used internally, and the packages that hold scripts and utilities for use on our servers. Access to this repository was restricted to our server IP space (more or less).

We occasionally did local backports of software and uploaded them to debian-local when we needed to use a backported version, but since I had upload rights to backports.debian.org, I more frequently just uploaded the backport there and then used pinning to pull the packages from there. At sites where there isn't a Debian Developer on staff, I would expect more use of the local repository to hold local backports.

In both repositories, we followed a distribution layout matching that of the Debian archive: unstable, testing, stable, and oldstable, identified by codename, with symlinks that change for each release. We pointed all of our servers to the repositories by codename so that, when a new version was released, they didn't automatically pick up the new packages. We automatically copied into testing every package uploaded to unstable rather than waiting some period of time like Debian does, since usually we had very few systems running unstable or testing. We generally uploaded packages to unstable and then pulled them or backported them into stable, although sometimes if we were lazy we'd upload directly to stable. We tried to use unstable first, though, for one of the same reasons that Debian does: it means the upgrade path when the next stable releases is taken care of.

In the debian-local repository, we also created some separate distributions for particular servers. For various reasons, we had to use our own packages for OpenLDAP on our directory servers and for Postfix on our mail routers, but we didn't want to use our customized packages on all of our servers for things like the regular OpenLDAP libraries. We therefore had distributions named stable-email and stable-ldap (actually, named by the codenames with symlinks for those names) and uploaded our customized packages only to those repositories.

Although we had some Ubuntu systems, we didn't maintain packages for specific Ubuntu distributions. Instead, we pointed our Ubuntu systems at the "closest" Debian distribution for that Ubuntu release. That mostly worked okay.

Configuration

Both of our repositories were managed by reprepro, with which we were extremely happy. They were completely separate from each other, including configuration, temporary directories, and the entire rest of the structure. They did, however, share a repository signing key.

A typical stanza in our distributions configuration file looked like this:

    Codename: squeeze
    Suite: stable
    Origin: Stanford
    Label: Debian
    Description: Stanford supplemental Debian repository
    Architectures: amd64 i386 source
    Components: main contrib non-free
    Pull: auto-stable
    SignWith: FDF37CD4279D4962
    DebIndices: Packages Release . .gz .bz2
    DscIndices: Sources Release .gz .bz2
    Contents: .gz .bz2
    Tracking: all
    Log:
     --changes /usr/bin/reprepro-notify

(Obviously this is from when squeeze was stable.) This is mostly obvious. The Pull setting will be discussed in a moment. We enableed source package tracking and have been very happy with it. /usr/bin/reprepro-notify is a very simple shell script that mails the *.changes file for a new upload to an internal reporting address.

We used the following pulls configuration file:

    Name: auto-stable
    From: sid
    FilterList: hold pull-stable

    Name: auto-oldstable
    From: sid
    FilterList: hold pull-oldstable

    Name: everything
    From: sid

The everything pull was used for testing. The others were for stable and oldstable, obviously, and reference files that define which packages to pull. For internal packages that bundled together scripts for a particular service, we found it easiest to automatically pull that package into stable whenever it was uploaded.

Finally, our incoming configuration:

    Name: default
    IncomingDir: /srv/repos/stanford/incoming
    TempDir: /srv/repos/stanford/tmp
    Allow: oldstable>lenny stable>squeeze unstable>sid
    Multiple: yes

This is for debian-stanford; the version for debian-local was similar but had more Allow mappings for our service-specific repositories.

Distribution Upgrades

When a new stable version was released, we did basically the same thing that Debian does:

  1. Update the distributions file to no longer pull everything into the previously testing, now stable distribution and switch it over to the auto-stable rule.

  2. Change the now-oldstable distribution over to the auto-oldstable rule.

  3. Change the suite configuration in the distributions file to move the stable and oldstable names.

  4. Create a new distribution for the new testing, configure it with the everything pull rule, and then do a reprepro pull to populate it from unstable.

It's important to point all systems at the distribution codename rather than directly at stable or oldstable, since otherwise the available packages can change unexpectedly without any configuration changes after the release of a new version of Debian.

Remote API

All of the staff making uploads also had root access on the repository server and could use reprepro commands directly, but that requires configuring the location of the GnuPG keyring used to sign the repository metadata and selecting the right repository to work with. It also requires logging on to the system, and we tried to avoid logging on to servers wherever possible in favor of exposing useful APIs via remctl. Finally, many of the reprepro commands should only be used by experts or should not normally be used in our repository.

We therefore wrote a wrapper around reprepro that adds an additional argument before each command to specify which repository to act on, can be run via remctl, and which sets up the location of the GnuPG keyring. That wrapper initially only exposed the commands copy, copysrc, list, ls, remove, and removesrc currently, and only added new commands when we had regular need of them.

The wrapper also added a new command, clean, which removed any files left behind in the incoming directory, so that people can remotely clean up after failed uploads that were rejected for some reason.

The wrapper isn't currently available outside of Stanford, but will be in the future (as soon as I have time to clean it up a bit and adapt it for my personal repository).

We also had a separate script that ran processincoming and pull after a new upload. Eventually, this will be rolled into the same wrapper.

Audits

When I left Stanford, I'd only done some preliminary work on audits to check repositories for consistency and sanity. At the time I left, there were only three audits, all of which were implemented by our wrapper and which were built on top of the reprepro commands plus some other standard Debian tools like rmadison:

build-needing

Shows the list of packages that aren't built for all supported architectures. This just runs reprepro build-needing, except that it does so for each codename and each architecture as one command.

multiple

Lists packages that are present in both debian-local and debian-stanford. Generally this is a mistake caused by someone being too used to uploading to local and forgetting to upload to stanford something that is used by campus.

obsolete

Lists packages whose version is older than the package in the same Debian distribution, as determined by rmadison. We use this to locate cases where we've locally packaged something that's now in Debian or backported something for which a newer version is now available in the stable distribution.

Last modified and spun 2015-11-01