Using Git for Debian Packaging

Table of Contents

Introduction
Initial Configuration
Repository Layout
Creating a New Package
Packaging Workflow
Handling Repacked Upstream
Backporting
Debian and Upstream Combined
Git Cheat Sheet
Converting to Git

Introduction

There are a bunch of other pages out there on the web, and this doesn't attempt to be a comprehensive guide. Instead, this is an opinionated description of the tools I personally use for Debian packaging with Git, and a description of my workflow. I maintain it partly for my own purposes as a reminder, but you may find it interesting while developing your own workflows.

The Git version control system (VCS) is a distributed VCS originally written by Linus Torvalds for Linux kernel development after moving away from BitKeeper. "Distributed" means that it has strong support for branching, merging, and parallel lines of development, stores the complete repository history locally, and is designed for multiple people working on the same software in parallel and merging in complex ways between their separate trees.

Of the distributed version control systems, Git has the strongest following among free software developers at the moment and is the most popular in Debian for packaging. Before switching to Git, I used Arch (bazaar and tla), bazaar-ng (bzr), and svk, as well as (non-distributed) Subversion. Prior to Debian packaging, I've also used CVS and RCS extensively, and I've used Mercurial at work (although not for Debian packaging). Disclaimer: I think Git is intuitive and natural, and I think Mercurial is awful, confusing, broken, and slow. If this is the exact opposite of your opinion (and that is true for many people), this workflow may not be very comfortable for you.

I used to have getting-started information about how to use Git on this page, but there are now so many resources on-line for learning Git that it didn't seem worthwhile to keep this up-to-date. One starting place if you're unfamiliar with Git basics is the Git user manual. There are lots of others.

Initial Configuration

First, you will need a build environment for packages. For new systems, I use sbuild with btrfs, documented on a separate page. The other common option is pbuilder, with or without cowbuilder to speed up the chroot setup.

There are a lot of different tools for using Git for Debian packaging with different properties. I won't try to present a comprehensive survey, just the option I use. I do most development with git-buildpackage, but use dgit to make the final upload.

First, set up some global configuration for git-buildpackage. I use the following settings in ~/.gbp.conf:

    [DEFAULT]
    builder = sbuild
    pristine-tar = True

    [buildpackage]
    export-dir = ../build-area
    tarball-dir = ../tarballs

    [import-orig]
    dch = False

    [dch]
    full = True

This enables pristine-tar, which stores information used to re-create the upstream tarball in the Git repository. Use of this program is a bit controversial since it doesn't always work properly, but I've had good luck with it. Just be aware that, if you run into problems, you may need to download the previous upstream tarball from Debian.

This uses the convention of ../build-area for storing build results (and ../tarballs for upstream tarballs if you're not using pristine-tar), rather than cluttering up the parent directory. I much prefer this, since Debian package builds generate a bunch of messy files, but be aware that dgit doesn't support this. That's why I use git-buildpackage for all the intermediate builds and only use dgit for the final upload build.

Repository Layout

For all Debian packages that aren't maintained as part of a larger packaging group with different conventions, I use the DEP-14 branch layout. The short version is that packaging for the unstable distribution goes in debian/master, packaging for stable versions (security updates, for instance) go in debian/<codename> branches named for the release code name, and upstream tarballs are imported into upstream/latest.

To indicate that a package uses this branch layout, I put the following in debian/gbp.conf in the package:

    [DEFAULT]
    debian-branch = debian/master
    upstream-branch = upstream/latest

This is included in the package, not just set locally in my own configuration, so that anyone who clones the package from its Git repository will get the proper configuration.

There are various options if Debian packaging requires changes to the upstream source. I've eventually converged on using the gbp pq tool (included in git-buildpackage) to manage a quilt series of patches against the upstream source while using Git to edit the packages when needed. More on that below. This means there are no persistent branches for patches against upstream, just transient branches named things like patch-queue/debian/master.

If upstream also uses Git, I add the upstream repository as a Git remote (usually using the remote name upstream) and will sometimes maintain a local master branch tracking upstream. One special case of this is when I'm also upstream, which is discussed below. One of the advantages of the DEP-14 branch layout is that it easily allows mixing upstream development and Debian patching in the same repository without mixing Debian packaging files with upstream files.

Creating a New Package

git-buildpackage can do most of the work of managing the upstream/latest branch. To start a new Git repository from an existing package without caring about the history, run:

    gbp import-dsc --git-debian-branch=debian/master \
        --git-upstream-branch=upstream/latest /path/to/dsc

You'll get a sub-directory of your current directory named after the package and containing an initialized Git repository with an upstream/latest and debian/master branch and with (if you use the above configuration) a pristine-tar branch that holds the metadata used by pristine-tar. gbp import-dsc will also create an upstream/* tag for the upstream version and a debian/* tag for the Debian version of the package.

If you're importing an existing package with source format 1.0 that has upstream patches mingled with Debian changes, you may need to shuffle things around a bit. I generally do that by saving off the upstream patches, reverting all those changes on debian/master, and then using gbp pq to re-apply the patches in a proper quilt sequence using the instructions below.

If you're instead starting from scratch with packaging a new package, create an empty directory, cd into it, and run git init to create an empty repository. Then rename the upstream tarball to <package>_<version>.orig.tar.gz (don't repack it unless you have to, just rename it) and run:

    gbp import-orig --git-debian-branch=debian/master \
        --git-upstream-branch=upstream/latest /path/to/upstream/tarball

This will create an upstream/latest branch and load the upstream source and then merge that onto your master branch. You can then create the debian directory and start creating the packaging.

I always use 3.0 (quilt) as the source package format (configured in debian/source/format). Maintaining changes from upstream as a stack of diffs is a bit tedious, but gbp pq makes it simple enough (described below), and it's very nice to have separated patches to send upstream when needed.

I also always write a debian/watch file if upstream puts their releases anywhere that uscan can work with (which is almost always). This not only allows Debian QA tools to alert me for new upstream releases, but also allows git-buildpackage to easily download the new upstream releases for me. (See below.) See the uscan(1) manual page for information on how to write that configuration file.

Packaging Workflow

Further Debian-specific work can then be done on the debian/master branch by switching to it with git checkout, modifying and committing things.

If I need to make changes to the upstream source, I first run:

    gbp pq import

to import the current patches and switch to a patch-queue branch. I then create new commits, and possibly rebase (against debian/master) to reorder or change existing commits. When done, I run:

    gbp pq export
    gbp pq drop
    git add debian/patches
    git commit

to commit the changes to the patch sequence.

When ready for a test build, run gbp buildpackage. The package will be built and, if successful, put in ../build-area, where I inspect the package with debc and run lintian on it to look for any issues.

When a new release is building and tested and ready to upload to Debian, I make sure that the target distribution in debian/changelog is correct and the timestamp on the changelog entry has been updated, and then run:

    origtargz
    dgit --gbp sbuild
    dgit --gbp push

to build the final version of the package. origtargz checks out the upstream tarball using pristine-tar and puts in the parent directory where sbuild and dgit expect it. Unfortunately, dgit gbp-build doesn't work because dgit doesn't support ../build-area and can't find the correct files.

When there's a new upstream release, run:

    git fetch upstream
    gbp pq import  # only if you have patches against upstream
    git checkout debian/master
    gbp import-orig --uscan --upstream-vcs-tag=<upstream-tag>
    # The rest are only if you have patches against upstream
    gbp pq rebase
    gbp pq export
    gbp pq drop
    git add debian/patches
    git commit

If you don't have a working debian/watch file or don't want to use uscan, you can also download the upstream release tarball, omit the --uscan flag to gbp import-orig, and just provide the path to the upstream release tarball. I do this when packaging software for which I'm also upstream, since I generally prepare the Debian package with a local release tarball that I haven't published yet, and then publish the release tarball and the Debian package at the same time.

This assumes that upstream is also using Git and tags their releases. If not, remove the git fetch command and the --upstream-vcs-tag flag. This fetches the upstream tarball using uscan, merges it into debian/master, and sets up that commit to merge with the upstream Git history, if available, so that you can easily look through upstream changes with git log.

If upstream always uses the same tag pattern for release tags, you can set that pattern with an upstream-vcs-tag setting in debian/gbp.conf and then don't have to remember to specify the flag. See the gbp-import-orig(1) man page for more details.

Note the gbp pq commands, which are necessary to rebase any patches you have against the upstream source. You can omit them if you're not carrying any patches.

When making additional changes to the Debian packaging, I still commit debian/changelog entries along with the corresponding change and just resolve the conflicts when I cherry-pick or merge. This doesn't really bother me, but it is some additional work. Some people instead leave the Debian changelog file alone until the release and then use gbp dch or some similar tool to generate it and then edit it. At some point, I'd like to play with that idea.

debcommit's support for determining the commit message from my changelog entry doesn't do what I want, so I write the commit message separately. Your mileage may vary.

gitk is incredibly useful for performing any sort of archaeology. It's particularly useful if you forget to tag uploads and need to figure out what specific revision corresponded to a Debian package upload.

Handling Repacked Upstream

If the upstream source has to be repacked for DFSG reasons, you can just generate a new *.orig.tar.xz file outside of Git and then import it as described above. However, Git and git-buildpackage have some additional features that may make this easier if you're just excluding certain files from the upstream source.

The easiest approach is when you use a copyright-format 1.0 debian/copyright file and have a working debian/watch file. In this case, add a Files-Excluded field to the first stanza of that file, listing the files in the upstream source that must be omitted. uscan will then automatically do the repacking for you. See uscan(1) for more information.

If you're not using that format, you can import the upstream distribution selectively using:

    gbp import-orig --upstream-vcs-tag=<upstream-tag> --filter='rfc*' \
        /path/to/upstream/tarball

for example, adding --filter options for every file in the upstream source that you need to exclude. (These options can be recorded in debian/gbp.conf for others working on the package rather than giving them directly on the command line.) Then, you can generate the DFSG-free version of the upstream tarball with:

    git archive --prefix=<package>_<version>.orig/ upstream/<version> \
        | xz > <package>_<version>.orig.tar.xz
    pristine-tar commit <package>_<version>.orig.tar.xz
    rm <package>_<version>.orig.tar.xz

Whether to use the uscan(1) approach, this approach, or the more traditional approach of generating the *.orig.tar.xz file separately and then using gbp import-orig as always is up to you and depends on which approach you find simpler.

Backporting

Often backporting a package to stable is as simple as just adding a new changelog entry for the backport and then rebuilding it in a stable chroot instead of an unstable chroot. This document doesn't cover that case, since it can be done without any representation in the version control repository (and normally there's no point in committing such backports anywhere).

If backporting requires more work, such as changing build dependencies or adjusting the packaging for tools that aren't present in stable, the easiest way to do this using Git is to create a new branch off of debian/master and make the necessary changes there. For example, for a stretch backport:

    git checkout -b debian/stretch-backports debian/master
    # edit debian/control and other files as needed
    # edit debian/changelog to add backport log entry
    gbp buildpackage --git-debian-branch=debian/stretch -v<last-version>

The -v option should specify the last version of the package available in stable, if it was previously in stable, so that the *.changes file will include all changes since that date. The version of the backport should follow the conventions of whatever repository for which it is being prepared; for the backports.debian.org repository, see the documentation for contributing.

For later backports, you normally will be able to merge the new master branch into the backport branch with:

    git checkout debian/stretch
    git merge debian/master

The debian/changelog file may have conflicts that have to be resolved, normally by removing the old backport log entry and adding a new one for the new backport.

Debian and Upstream Combined

For many of my packages, I'm both the Debian packager and the upstream maintainer. In those cases, I want a combined repository that I can use for both upstream maintenance and for Debian packaging.

Before using Git, I used to do this by maintaining the debian directory in with the upstream code like any other part of the code, on the main development branch, and just excluding it from releases. Then, to build Debian packages, I'd unpack the latest tarball, export the debian directory from the repository into the unpacked working tree, and build the result. I had a script that automated much of this process.

This works, but it has a few drawbacks:

After switching to Git, I wanted a more general solution without these limitations that could also leverage pristine-tar so that all the information required to build the package was within the repository.

The wonderful part of the DEP-14 layout plus git-buildpackage's support for merging with upstream releases when importing upstream tarballs means that using the above workflow just works. I keep the upstream source on the master branch, the Debian packaging on the debian/master branch, generate a tarball as normal for upstream releases, and import it like I would with any other Git-using upstream. All the right things happen, including maintaining a proper merged Git history in the Debian packaging branch.

For upstream releases, I make release/<version> signed Git tags, and then reference those with --upstream-vcs-tag or, better, the corresponding configuration in debian/gbp.conf.

Converting an existing repository that combines Debian packaging and the upstream development is a bit tricky, since you want to create a new debian/master branch that shares the history of the current master and an upstream/latest branch that can be merged into debian/master and used by gbp import-orig, but you also want to remove the debian/* directory from the master branch. The first time I tried this, I ended up merging the removal of the debian/* directory into my debian/master branch and having to back out of that.

After some experimentation, the following approach seems to work the best:

  1. Before removing debian/* from master, create the debian/master branch with git branch debian/master master.

  2. git rm -r debian on the master branch and commit it.

  3. git branch upstream/latest master. Now you have an upstream branch with the right history and without the debian/* directory, although it doesn't match the upstream tarball release (it's missing generated files).

  4. Record in Git that the debian/master branch is already a correct reflection of the changes relative to the upstream/latest branch by adding a merge commit. git checkout debian/master and then git merge -s ours upstream/latest. The -s ours is the key trick; it records the merge commit without deleting debian/* as part of the merge.

  5. Now you can run gbp import-orig on the current upstream release tarball, let it fix up the upstream branch to match the release tarball by adding all the generated files, and then merge them into the debian/master branch. You can do this now if you haven't made any changes since the previous release. If you have made changes, the easiest thing to do is to wait until you do another release and then have that be the first true imported release.

I don't bother to go back and try to reconstruct history and put previous releases in the upstream/latest branch. It's a lot of work and doesn't seem worth the effort.

Git Cheat Sheet

Most of the common Git commands will quickly enter your finger memory if you've used Git for very long. Here are a few less obvious ones that are worth knowing.

git commit --amend

Modify the last commit. I use this all the time, usually because I commit and then remember I need to fix one more thing, or made an error in the commit message. This is a form of history rewriting, so like all history rewriting should be done before you push.

git rebase -i <commit>

Do an interactive rebase of your current branch against some other commit (usually the name of a branch, but sometimes it's useful to rebase on top of a specific commit). This means Git will find the common ancestor, and then interactively let you rewrite the commits as if they were applied directly on top of that commit. You'll be given an editor window into which you can enter commands that can do a lot of different things: combine commits (the most common thing I do), reorder them, change their commit messages, or modify them.

git add -p <file>

Prompts you for each modified hunk of a file and asks you whether to add it to the index (stage it for the next commit). This is very useful if you have a dirty tree with work that should be in two or more separate patches, since you can select some modifications of a file to be in the next commit while leaving other ones uncommitted.

I also add the following aliases to my personal ~/.gitconfig:

    [alias]
            update = pull --rebase --stat
            where = symbolic-ref HEAD

This lets me run git update to do a git pull --rebase and quickly rebase local work on top of upstream, something I do constantly throughout the day. git where shows the current branch extremely quickly, without doing the slow checks for uncommitted files that git status does.

Converting to Git

Subversion to Git Migration

I had a bunch of old Debian packaging repositories in Subversion that I had to convert to Git. There are a lot of different ways to do this, but here's the one that works for me. This technique is heavily based on a blog entry from Martin Krafft. I've done this conversion, so am not updating these instructions, but they probably haven't changed.

First, install git-svn if it's not already installed.

My packages all used an svn-buildpackage layout, which meant there was an upstream directory under branches that had the imports of the upstream source. I'm usually willing to throw away any other branches, so I started with:

    git svn clone --stdlayout --branches=branches/upstream --no-metadata \
        -A /path/to/authors/file <svn-url>

This will create a new sub-directory named after the last component of the <svn-url> with a new Git repository. This command treats all the upstream tags and current as branches and ignores the other branches; if you want to keep all the branches, svn mv the contents of branches/upstream into branches first and then point git svn at the whole branches directory.

Note the -A option; use that to specify the path to a mapping from usernames in the Subversion commits to full names and e-mail addresses for Git. Without this, your old commits will have ugly identification. See the git-svn man page for the details on the format.

--no-metadata means that we're doing a one-way conversion and git svn shouldn't keep the data required to support commits back to the Subversion repository.

git svn will leave your Subversion trunk and branches as remote branches in your repository and will create all of your tags as more remote branches. The first step is to clean that up. Get a full branch list with git branch -r. Then, create new local branches for the remote branches that are really branches; at the least, you'll probably want:

    git branch upstream/latest branches/current
    git branch -r -d branches/current

Note the removal of the remote tracking branch using git branch -r -d; that's how you get rid of those. Next, convert each tag into a real tag. The revision to which the tag should apply will be the revision prior to the one recorded in the repository, so take advantage of Git's ^ shortcut for previous revision with commands like:

    git tag debian/<version> tags/<version>^
    git branch -r -d tags/<version>

Similarly for all the tags of the upstream imports, do:

    git tag upstream/<version> branches/<version>^
    git branch -r -d branches/<version>

Once you finish this, git branch -r should show no remote tracking branches and only your regular upstream and master branches.

If you used the svn-buildpackage mergeWithUpstream feature to not store the upstream source in your Git repository, the conversion has a few more steps. You need to import the upstream source into the repository to use Git the way that I outline in this document. You should be able to do that by just importing the most recent upstream tarball from Debian with gbp import-orig. This will commit and tag the upstream source and merge it into your master branch, fleshing out your repository. All of your old tags will still have only modified files, but normally that's not a serious problem.

If you had to do gbp import-orig as described above, it will take care of committing the pristine-tar metadata. Otherwise, grab the last upstream orig.tar.gz (and make sure that it's named properly), put it somewhere not in the current directory, and run:

    pristine-tar commit /path/to/orig.tar.gz

to load it into the repository. If you want, you can also load older upstream tar.gz files, but I never bother.

You can now delete some additional junk left over from the git svn conversion process. Edit .git/config and remove the information about the remote tracking branches for the Subversion repository, and delete all of .git/svn.

If you're like me, you weren't very good about always tagging releases in Subversion. You can now much more easily fix this by browsing the repository using gitk and finding the release points for each Debian package. When you do, paste the hash (shown nicely in gitk for you) into a git tag command like:

    git tag debian/<version> <hash>

You can add the -s flag to create signed tags.

When you select a revision in gitk, it copies the hash of that revision into the X cut buffer for you, so you can paste it into a command without selecting it.

Finally, do the same things as noted above under setting up a Debian packaging repository (clone master to debian/master, move upstream changes into a patch series, and so forth).

CVS to Git Migration

My old CVS repositories don't use branches or tags, so importing them was much easier and essentially follows the git-cvsimport man page.

First, install the git-cvs package if you haven't already.

Now, run git cvsimport. Since I'm doing one-way conversions and plan on discarding all the old CVS information afterwards, I use options like:

    git cvsimport -d <cvs-repository> -o master -k -A <authors-file> \
        -a -C <project> <module>

where <project> is the name of the new directory you want to create with the imported repository. As with Subversion, you'll need an authors file that maps the usernames stored in CVS commit messages to identities in the new Git repository.

If you're like me, you'll then want to go back through the history with gitk and tag all the past releases with git tag.

bzr to Git Migration

When I did a Google search on migrating a bzr repository to Git, most of the documentation said to use tailor. However, it didn't work; it got as far as a changeset that renamed ChangeLog to CHANGES.old and then bailed because it claimed the git add command on CHANGES.old failed. When I ran exactly the command that it claimed to have ran by hand, it worked fine, but I couldn't figure out how to convince tailor of that.

Thankfully, there's a better alternative, although it's not well-documented yet. Git and bzr both have support for the fast-import repository serialization format, although the bzr support is somewhat hidden away if you're using Debian. Here are the instructions that I used back when I did this; now, it looks like bzr-fastimport is a regular Debian package and installs a plugin, so I suspect that just installing it and using the commands it provides will work.

First, get the bzr-fastimport source tree from Launchpad:

    bzr branch lp:bzr-fastimport

Then, create a new directory and empty Git repository for your project:

    mkdir <project>
    cd <project>
    git init

Now, you can import your old bzr repository by serializing it using the bzr-fast-export.py script that comes with bzr-fastimport:

    /path/to/bzr-fastimport/exporters/bzr-fast-export.py \
        /path/to/bzr-repository | git fast-import
    git checkout master

This will even preserve tags in your bzr repository.

Last spun 2018-02-10 from thread modified 2018-02-08