Hacking INN

Table of Contents

  1. Hacking INN
  2. Configuring and Portability
  3. Autobuilds
  4. Documentation
  5. Error Handling
  6. Test Suite
  7. Makefiles
  8. Scripts
  9. Include Files
  10. Libraries
  11. Coding Style
  12. Making a Release
  13. References

Hacking INN

This file is for people who are interested in making modifications to INN. Normal users can safely skip reading it. It is intended primarily as a guide, resource, and accumulation of tips for maintainers and contributors, and secondarily as documentation of some of INN's internals.

First of all, if you plan on working on INN source, please start from the current development tree. There may be significant changes from the previous full release, so starting from development sources will make it considerably easier to integrate your work. You can check out or clone the current development tree using Git from:

    <https://github.com/InterNetNews/inn.git>

or browse the current development source on Github at:

    <https://github.com/InterNetNews/inn>

You will need autoconf 2.64 or later to use the development tree. After checking out the tree, run ./autogen to generate the necessary autoconf files.

Pull requests must complete an automatic build/test cycle with warnings enabled. You can run an equivalent build with ./ci/test.

Daily snapshots can be found at:

    <https://ftp.isc.org/isc/inn/snapshots/>

Configuring and Portability

All INN code should be written expecting ANSI C and POSIX. There is no need to attempt to support pre-ANSI compilers, and ANSI-only features such as <stdarg.h>, string concatenation, #elif, and token pasting may be used freely. So far as possible, INN is written to attempt to be portable to any system new enough that someone is likely to want to run a news server on it, but whenever possible this portability should be provided by checking for standard behavior in configure and supplying replacements for standard functions that are missing.

When there is a conflict between ANSI C and C99, INN code should be written expecting C99 and autoconf used to patch up the differences.

Try to avoid using #ifdef and the like in the middle of code as much as possible. Instead, try to isolate the necessary portability bits and include them in libinn or at least in conditional macros separate from the code. Trying to read code littered with conditional compilation directives is much more difficult.

The shell script configure at the top level of the source tree is generated by autoconf from configure.ac and the additional macros in the m4 directory, and include/config.h.in is generated by autoheader from configure.ac. At configure time, configure generates include/config.h and several other files based on options it was given and what it discovers about the target system.

All modifications to configure should instead be made to configure.ac. The autoconf manual (available using info autoconf if you have autoconf and the GNU info utilities installed on your system) is a valuable reference when making any modifications.

To regenerate configure, just run autoconf. To regenerate include/config.h.in, run autoheader. Alternately, to regenerate both, you can run ./autogen. Please don't include patches to either configure or include/config.h.in when sending patches to INN; instead, note in your patch that those files must be regenerated. These (and all other) generated files should not be checked into Git.

At the time of this writing, autoconf 2.64 or later is required.

The supporting files for autoconf are in the support subdirectory, including the files config.guess and config.sub to determine the system name and ltmain.sh for Libtool support. The latter file comes from the Debian package of Libtool, available from <https://packages.debian.org/libtool> (ltmain.sh is generated in the build-aux subdirectory after building the package with for instance debuild -us -uc); the canonical versions of the former two files are available from <https://ftp.gnu.org/gnu/config/> (which currently redirects to <https://git.savannah.gnu.org/cgit/config.git/tree/>).

In addition, m4/libtool.m4 and a few other m4 files used by INN are just a copy of the corresponding files in the m4 subdirectory of the Debian package of the Libtool distribution. (Using Libtool without using automake requires a few odd hacks.) New versions should be checked in periodically when available. There are no INN-specific modifications to those files except for ltmain.sh which recognizes the additional -S flag that INN's install-sh script uses, and a fix for a proper handling of circular dependencies.

This install-sh script should also be updated at the same time from <https://git.savannah.gnu.org/cgit/automake.git/tree/>; it similarly contains a local modification mentioned in a comment at the beginning of the file.

Finally, the m4/pkg.m4 macro should be updated from <https://gitlab.freedesktop.org/pkg-config/pkg-config/>.

Autobuilds

Automatic build logs on several platforms are available at <http://usenet.trigofacile.com/autobuild/inn/>.

Documentation

INN's documentation is currently somewhat in a state of flux. Much of the documentation is still in the form of man pages written directly in nroff. Some parts of the documentation have been rewritten in POD; that documentation can be found in doc/pod. The canonical source for most of the text documentation, including README, INSTALL, NEWS, and this file, is also POD.

If you're modifying some part of INN's documentation and see that it has a POD version in doc/pod, you should make the modifications to the POD source and then regenerate the derived files. For a quick introduction to POD, see the perlpod(1) man page on your system (it should be installed if you have Perl installed).

When writing new documentation, write in whatever format you care to; if necessary, we can always convert it to POD or whatever else we want to use. Having the documentation exist in some form is more important than what language you write it in. If you don't have a preference, however, we prefer new documentation to be in POD.

If you use POD or regenerate POD documentation, please install something close to the latest versions of the POD processing utilities to avoid changes to the documentation depending on who generated it last. You can find the latest version of the podlators distribution on CPAN (<https://metacpan.org/dist/podlators>). For versions of Perl before 5.6.1 (as Perl 5.6.1 and later come with a recent enough version), you'll also need Pod::Parser (see <https://metacpan.org/pod/Pod::Parser>).

There are makefile rules in doc/pod/Makefile to build all of the documentation whose master form is POD; if you add additional documentation, please add a rule there as well. Documentation should be generated by cd'ing to doc/pod and typing make file where file is the relative path to the documentation file. This will get all of the various flags right for pod2text or pod2man.

Error Handling

INN has a set of generic error handling routines that should be used as much as possible so that the same syntax can be used for reporting errors everywhere in INN. The six basic functions are notice, sysnotice, warn, syswarn, die, and sysdie; notice prints or logs an informative message, warn prints or logs a warning, and die does the same and then exits the current program. The sys* versions add a colon, a space, and the value of strerror(errno) to the end of the message, and should be used to report failing system calls.

All of the actual error reporting is done via error handlers, and a program can register its own handlers in addition to or instead of the default one. The default error handler (message_log_stderr) prints to stderr, prepending the value of message_program_name if it's set to something other than NULL. Several other error handlers are available, particularly including ones that use syslog. See include/inn/messages.h for all of the available options.

There is a different set of error handlers for notice/sysnotice, warn/syswarn, and die/sysdie. To set them, make calls like:

    message_handlers_warn(1, message_log_stderr);
    message_handlers_die(2, message_log_stderr, message_log_syslog_err);

The first argument is the number of handlers, and the remaining arguments are pointers to functions taking an int (the length of the formatted message), a const char * (the format), a va_list (the arguments), and an int that's 0 if warn or die was called and equal to the value of errno if syswarn or sysdie was called. The length of the formatted message is obtained by calling vsnprintf with the provided format and arguments, and therefore is reliable to use as the size of a buffer to malloc to hold the result of formatting the message provided that vsnprintf is used to format it (warning: the system vsprintf may produce more output under some circumstances, so always use vsnprintf).

The error handler can do anything it wishes; each error handler is called in the sequence given. Error handlers shouldn't call warn or die unless great caution is taken to prevent infinite recursion. Also be aware that sysdie is called if malloc fails in xmalloc, so if the error handler needs to allocate memory, it must not use xmalloc or a related function to do so and it must not call die to report failure. The default syslog handlers report memory allocation failure to stderr and exit.

Finally, die and sysdie support an additional handler (the external variable message_fatal_cleanup) that's called immediate before exiting, takes no arguments, and returns an int which is used as the argument for exit. It can do any necessary global cleanup, call abort instead to generate a core dump or the like.

The advantage of using this system everywhere in INN is that library code can use notice, warn, and die to report messages and errors, and each calling program can set up the handlers as appropriate to make sure the errors go to the right place. The default handler is fine for interactive programs; for programs that run from interactive scripts, adding something like:

    message_program_name = "program";

to the beginning of main (where program is the name of the program) will make it easier to figure out which program the script calls is failing. For programs that may also be called non-interactively, like rnews, one may want to set up handlers like:

    message_handlers_notice(2, message_log_stdout, message_log_syslog_info);
    message_handlers_warn(2, message_log_stderr, message_log_syslog_warning);
    message_handlers_die(2, message_log_stderr, message_log_syslog_err);

Finally, for daemons and other non-interactive programs, one may want to do:

    message_handlers_notice(1, message_log_syslog_info);
    message_handlers_warn(1, message_log_syslog_warning);
    message_handlers_die(1, message_log_syslog_err);

to report errors only via syslog. (Note that if you use syslog error handlers, the program should call openlog first thing to make sure they are logged with the right facility.)

For historical reasons, error messages that are fatal to the news subsystem are logged at the LOG_CRIT priority, and therefore die in innd should use message_log_syslog_crit. This seems like priority inflation and may change in the future to message_log_syslog_err.

Test Suite

The test suite for INN is located in the tests directory and is just getting started. The test suite consists of a set of programs listed in tests/TESTS and the scaffolding in the runtests program.

Adding new tests is very straightforward and very flexible. Just write a program that tests some part of INN, put it in a directory under tests named after the part of INN it's testing, and have it output first a line containing the count of test cases in that file, and then for each test a line saying ok n or not ok n where n is the test case number. (If a test is skipped for some reason, such as a test of an optional feature that wasn't compiled into INN, the test program should output ok n # skip.) Add any rules necessary to build the test to tests/Makefile (note that for simplicity it doesn't recurse into subdirectories) and make sure it creates an executable ending in .t. Then add the name of the test to tests/TESTS, without the .t ending.

For C tests, you probably want to use the functions in tests/tap/basic.c (prototypes in tests/tap/basic.h) to handle all of the output. For shell script tests, tests/tap/libtap.sh contains helpful shell functions. See the existing tests for some hints about how to write new tests, as well as the tests/README guide.

One naming convention: to distinguish more easily between for example lib/error.c (the implementation) and tests/lib/error-t.c (the test suite), we add -t to the end of the test file names. So tests/lib/error-t.c is the source that compiles into an executable tests/lib/error.t which is run by putting a line in tests/TESTS of just lib/error.

Note that tests don't have to be written in C; in fact, lib/xmalloc.t is just a shell script (that calls a supporting C program). Tests can be written in shell or Perl (but other languages should be avoided because someone who wants to run the test suite may not have it) and just have to follow the above output conventions.

Additions to the test suite, no matter how simple, are very welcome.

Running the test suite is done with:

    make check

A single test can be run with verbose ouput via:

    tests/runtests -o <name-of-test>

Using runtests ensures that necessary environment variables are set up.

Makefiles

All INN Makefiles include Makefile.global at the top level, and only that makefile is a configure substitution target. This has the disadvantage that configure's normal support for building in a tree outside of the source tree doesn't work, but it has the significant advantage of making configure run much faster and allowing one to run make in any subdirectory and pick up all the definitions and settings from the top level configuration.

All INN Makefiles should also set $(top) to be the path to the top of the build directory (usually relative). This path is used to find various programs like fixscript and Libtool so that the same macros (set in Makefile.global) can be used all over INN.

The format of INN's Makefiles is mostly standardized; the best examples of the format are probably frontends/Makefile and backends/Makefile, at least for directories with lots of separate programs. The ALL variable holds all the files that should be generated, EXTRA those additional files that were generated by configure, and SOURCES the C source files for generating tag information.

There are a set of standard installation commands defined in make variables by Makefile.global, and these should be used for all file installations. See the comment blocks in Makefile.global.in for information on what commands are available and when they should be used. There are also variables set for each of the installation directories that INN uses, for use in building the list of installed paths to files.

Each subdirectory makefile should have the targets all (the default), clean, clobber/distclean, install, and profiled. The profiled target generates a profiling version of the programs (although this hasn't been tested much). These rules should be present and empty in those directories where they don't apply.

Be sure to test compiling with both static and dynamic libraries and make sure that all the Libtool support works correctly. All linking steps, and the compile steps for all library source, should be done through $(LIBCC) and $(LIBLD) (which will be set as appropriate in Makefile.global).

Scripts

INN comes with and installs a large number of different scripts, both Bourne shell and Perl, and also comes with support for Tcl scripts (although it doesn't come with any). Shell variables containing both configure-time information and configuration information from inn.conf are set by the innshellvars support libraries, so the only system-specific configuration that should have to be done is fixing the right path to the interpreter and adding a line to load the appropriate innshellvars.

support/fixscript, built by configure, does this. It takes a .in file and generates the final script (removing the .in) by fixing the path to the interpreter on the first line and replacing the second line, whatever it is, with code to load the innshellvars appropriate for that interpreter. (If invoked with -i, it just fixes the interpreter path.)

Scripts should use innshellvars (via fixscript) to get the right path and the right variables whenever possible, rather than having configure substitute values in them. Any values needed at run-time should instead be available from all of the different innshellvars.

As for Perl, the INN::Config module has the same features as innshellvars.pl (only kept for compatibility reasons with old scripts not shipped with INN); however, it can be safely used with warnings on in Perl scripts.

See the existing scripts for examples of how this is done.

Include Files

Include files relevant to all of INN, or relevant to the two libraries built as part of INN (the utility libinn library and the libinnstorage library that contains all storage and overview functions) are found in the include directory; other include files relevant only to a portion of INN are found in the relevant directory.

Practically all INN source files will start with:

    #include "portable/system.h"

This header file includes config.h, which picks up all defines generated by autoconf and is necessary for types that may not be present on all systems (uid_t, pid_t, size_t, uint32_t, and the like). It therefore should be included before any other headers that use those types, as well as to get general configuration information. The config.h header also includes inn/macros.h, inn/options.h, inn/system.h and portable/stdbool.h which pick up additional support macros and compile-time configuration.

Besides, portable/system.h also includes all of the following headers, portably:

   #include <inttypes.h>
   #include <limits.h>
   #include <stdarg.h>
   #include <stdbool.h>
   #include <stddef.h>
   #include <stdint.h>
   #include <stdio.h>
   #include <stdlib.h>
   #include <string.h>
   #include <strings.h>
   #include <sys/types.h>
   #include <unistd.h>

except that it doesn't include headers that are missing on a given system, replaces functions not found on the system with the INN equivalents, provides macros that INN assumes are available but which weren't found, and defines some additional portability things. Even if this is more headers than the source file actually needs, it's generally better to just include portable/system.h rather than trying to duplicate the autoconf-driven hackery that it does to do things portably. The primary exception is for source files in lib that only define a single function and are used for portability; those may want to include only config.h so that they can be easily used in other projects that use autoconf. config.h is a fairly standard header name for this purpose.

There are portable wrappers around several header files that have known portability traps or that need some fixing up on some platforms. Look in include/portable and familiarize yourself with them and use them where appropriate.

Another frequently included header file is inn/libinn.h, which among other things defines xmalloc(), xrealloc(), xstrdup(), and xcalloc(), which are checked versions of the standard memory allocation routines that terminate the program if the memory allocation fails. These should generally always be used instead of the regular C versions. inn/libinn.h also provides various other utility functions that are frequently used.

inn/paths.h includes a wide variety of paths determined at configure time, both default paths to various parts of INN and paths to programs. Don't just use the default paths, though, if they're also configurable in inn.conf; instead, call innconf_read() and use the global innconf structure.

Other files in include are interfaces to particular bits of INN library functionality or are used for other purposes; see the comments in each file.

Eventually, the header files will be separated into installed header files and uninstalled header files; the latter are those headers that are used only for compiling INN and aren't useful for users of INN's libraries (such as innperl.h). All of the installed headers will live in include/inn and be installed in a subdirectory named inn in the configured include directory. This conversion is still in progress.

When writing header files, remember that C reserves all identifiers beginning with two underscores and all identifiers beginning with an underscore and a capital letter for the use of the implementation; don't use any identifiers with names like that. Additionally, any identifier beginning with an underscore and a lower-case letter is reserved in file scope, which means that such identifiers can only be used by INN for the name of structure members or function arguments in function prototypes.

Try to pay attention to the impact of a header file on the program namespace, particularly for installed header files in include/inn. All symbols defined by a header file should ideally begin with INN_, inn_, or some other unique prefix indicating the subsystem that symbol is part of, to avoid accidental conflicts with symbols defined by the program that uses that header file.

Libraries

INN includes three shared libraries, in lib, storage, and history. Whenever any of the source to those libraries is modified, the library version information must be updated at the top of the Makefile in that directory. Follow the instructions in the Libtool manual under Versioning / Updating version info, with the exception that, contrary to point 2, it's useful to those running snapshots to update the version information more frequently than for each public release. Remember that the version information has to be updated to allow recovery from a make update command.

Coding Style

INN has quite a variety of coding styles intermixed. As with all programs, it's preferable when making minor modifications to keep the coding style of the code you're modifying. In INN, that will vary by file. (Over time we're trying to standardize on one coding style, so changing the region you worked on to fit the general coding style is also acceptable).

When modifying or writing new code in C, it should conform to the Clang-format style of the project. Similarly, Perl, Python and shell scripts should conform to respectively the perltidy, Black and shfmt code styles of the project. Just run make reformat at the top level of the source tree to properly reformat all files.

If you're writing a substantial new piece of code, the prevailing "standard" INN coding style is the following:

For users of emacs cc-mode, use auto-formatting with clang-format and Black.

Finally, if possible, please don't use tabs in source files, since they can expand differently in different environments. In particular, please try not to use the mix of tabs and spaces that is the default in emacs. If you use emacs to edit INN code, you may want to put:

    ; Use only spaces when indenting or centering, no tabs.
    (setq-default indent-tabs-mode nil)

in your ~/.emacs file.

Note that this is only a rough guideline and the maintainers aren't style nazis; we're more interested in your code contribution than in how you write it.

Making a Release

This is a checklist that INN maintainers should go through when preparing a new release of INN.

  1. Update the files shipped with INN, and that are maintained by external projects.

  2. Monitor changes carried on a few other projects.

  3. Build INN (including the contrib and tests directories) with warnings on (make warnings) and run the test suite (make tests). Fix any errors.

    Several builds with different configure-time options should be tested, so as to be sure they work fine (especially --enable-shared=no, --enable-tagged-hash, --enable-keywords, --enable-largefiles and --enable-reduced-depends).

    Ideally, ensure INN builds fine with the latest versions of external libraries it has support for (Berkeley DB, blacklistd, Cyrus SASL, libcanlock, LibreSSL, MIT Kerberos v5, OpenSSL, Perl, Python, SQLite, zlib).

  4. Configure INN with all the external libraries it supports, and run make depend to ensure dependencies are properly defined in all INN Makefiles. Commit the changes, if appropriate.

  5. Ensure that the source code is properly formatted with the latest versions of reformatting programs (run make reformat).

  6. If possible, on a news server running the forthcoming release, run the following commands to make sure they do not produce any errors:

        cnfsstat -a -v
        ctlinnd checkfile
        inncheck -a --perm --pedantic
        ovdb_stat -klmMtv
        ovsqlite-util -A
        scanlogs norotate
        scanspool -n -v
        tdx-util -A
  7. Maintain up-to-date information in various files.

  8. If making a major release, create a new branch in Git named after the major release. This branch will be used for minor releases based on that major release and can be done a little while before the .0 release of that major release.

        git checkout -b Y.Y
        git push --set-upstream origin Y.Y

    Then, in that newly created branch, adapt the first paragraph of readme.pod, remove its second paragraph which deals with development versions, and update the URLs to point to the documentation of the new stable version.

  9. Check out a copy of the release branch, and perform the following actions.

  10. Run make release for a final release, support/mksnapshot BETA b1 for the first beta version of a new release, or support/mksnapshot RC rc1 for the first release candidate version of a new release.

  11. On the branch where make release was run, modify the VERSION_EXTRA variable in Makefile.global.in so that the prerelease value does not appear in the upcoming Git tag.

    When making the final release, this variable should be empty. For a testing release, it should reflect the version name used in the make release command (like rc1 version).

  12. Create an annotated signed tag corresponding to the tarball, using a PGP key declared in the committer's GitHub profile:

        git tag Y.Y.Y <commit-hash> -s -m "INN Y.Y.Y release tag"
        git push origin Y.Y.Y

    Check that it appears as Verified in the GitHub interface.

  13. Re-add the prerelease value to the VERSION_EXTRA variable in Makefile.global.in.

  14. Publish a release in GitHub, based on that tag. For a testing release, tick the box corresponding to a pre-release.

    Include the description from NEWS, and attach the tarball. The description should be modelled on what was done for previous releases.

    When publishing the final release, empty the now redundant description of the pre-release, and just refer to the final release.

  15. For a testing release, announce its availability in inn-workers, along with a draft of the release announcement for proof-reading. Install the testing release on at least one system and make sure that system runs fine for at least a few days.

    That's all for a testing release. The following steps only concern final releases.

  16. Bump revision numbers to reflect the one of the following release, especially in doc/pod/install.pod and readme.pod for major releases, configure.ac and Makefile.global.in for both minor and major releases. (It is not necessary to bump the revision number if the release is supposed to be the last one in a branch.)

  17. Update milestones in the GitHub bug tracker to reflect the new release.

  18. Make the previously generated release tar file available into the public area of ftp.isc.org. Sign the release, creating a .asc file, using:

        gpg --detach-sign --armor inn-Y.Y.Y.tar.gz

    Update the inn.tar.gz* links and possibly move older releases off into the OLD directory.

    (Currently, this is all done by Russ by updating the files in <https://archives.eyrie.org/software/inn/> and then letting ISC automatically mirror them via rsync within a day. The ISC web site does not need being updated as it does not directly link to the release, nor mention the version of the last release. Just check the information at <https://www.isc.org/othersoftware/#INN> is still available and accurate, though.)

  19. Now that the signature of the release has been generated, attach it to the GitHub release.

  20. After ftp.isc.org has correctly mirrored the release, send an announce on inn-announce and in news.software.nntp (with a possible crosspost to news.admin.announce).

  21. Ping Russ to clean up the snapshots generated for the previous version and, for major releases, update the branch used for the generation of STABLE snapshots.

  22. Ping Russ to update <https://www.eyrie.org/~eagle/software/inn/> with the latest version information and, for a major release, clone the documentation tree for CURRENT as a new stable documentation tree for the stable release series.

    Also ensure the URL redirection of the docs-Y.Y subdirectory corresponding to the next major release points to docs.

References

Some additional references that may be hard to find and may be of use to people working on INN:

<https://www.eyrie.org/~eagle/nntp/>

The home page for the IETF NNTP standardization effort, including links to the IETF NNTP working group archives and copies of the latest drafts of the new NNTP standard. The old archived mailing list traffic contains a lot of interesting discussion of why NNTP is the way it is.

<https://www.eyrie.org/~eagle/usefor/>

A collection of documents about the Usenet article format, including most of the relevant RFCs and Internet-Drafts.

<https://mailarchive.ietf.org/arch/browse/usefor/>

The archives for the USEFOR IETF working group, the working group for the RFC 1036 replacement (the format of Usenet articles), now published as RFC 5536 and RFC 5537.

<http://www.mibsoftware.com/userkt/userkt.html>

Forrest Cavalier provides several tools related to earlier versions of INN (2.0 to 2.3.x). His web site is a great source of information about Usenet in general.

<http://tools.ietf.org/html/draft-lutchann-ipv6-intro-00>

A primer on IPv6 with pointers to the appropriate places for more technical details as needed, useful when working on IPv6 support in INN.

Last modified and spun 2024-02-25