| Russ Allbery > FAQs and Documentation | Soundness for C News > |
Introduction and INN
by Russ Allbery <eagle@eyrie.org>
Last modified November 21, 2018
This is part one of a three part FAQ on maintaining a sound Usenet II site. This first part contains an introduction and overview, including pointers to more information on Usenet II, and then details on configuring INN for Usenet II soundness. The second part covers C News, and the third part covers other news servers (DNEWS, Typhoon, Cyclone, and Diablo) and the credits.
Usenet II, the net.* hierarchy, is based on the cooperation of sites who have agreed to enforce certain rules, for example limits on cross-posts and multi-posts. Usenet II is administered by a steering committee and a czar for each hierarchy within Usenet II. This FAQ contains specific information about how to set up a news server to carry Usenet II (the net.* hierarchy) in a sound manner, where sound is defined in the above-referenced rules. For more information, including who to contact to join Usenet II, please see the above web page.
Usenet II is slightly more work for a news admin to carry than a regular hierarchy. This FAQ is designed to minimize this work by laying out in precise detail how to make your server sound and keep it sound, and by mentioning the things you have to worry about and the things you don't have to worry about.
If you are or are considering becoming a Usenet II site, you need to be aware that remaining a sound Usenet II site is an ongoing process. Usenet II is based on the concepts of responsibility, accountability, and mutual cooperation. You will be expected to take action if one of your users repeatedly violates the rules, and you will be expected to keep your server sound through upgrades, new feeds, and potential unsound behavior by your peers. That being said, we want your site to be part of Usenet II and we want to help you remain sound. This document is part of that; another part of that is that we'll try to notify you of any problems, provide help with solutions, and otherwise offer you community support in maintaining your site.
This FAQ deals with the technical issues of maintaining a sound site. There are also policy issues, including how to keep the posts from your users sound and what to do about users who violate the rules, but hints and suggestions on how to do that are beyond the scope of this FAQ except insofar as they can be enforced via technical means. In general, the Usenet II community does not care how you remain sound provided that you do; no particular policy stance is required on your part provided that your site doesn't generate unsound posts.
Usenet II soundness has two components: Do not originate unsound posts, and do not inject unsound posts into Usenet II. The main types of unsound posts you have to worry about are posts from unsound sites and posts with incorrect distributions. The remainder, for the time being, are being left to the net.monitor and other similar tools, since INN and other major news servers cannot handle them directly.
It's also desirable, although not required, to avoid sending posts from Usenet II to unsound sites.
Propagating a sound post to an unsound site is called a leak. Sending an unsound post to a sound site is called a hole. Holes are actually against the Usenet II rules; leaks are just something that's nice to avoid.
The net.monitor, referred to several times below, is a program that runs as a program feed and removes unsound posts (currently with cancels, but better methods will be used in time). Eventually, the net.monitor (or some equivalent in Perl filters or the like) should ideally be running at a reasonable percentage of sites carrying net.*.
Following are the details of setting up and maintaining a sound server with INN. For details on other servers, see the additional parts of this FAQ.
No patches to INN are required to maintain a sound site, just configuration changes. One patch is recommended, though, for versions of INN earlier than 2.0, and not applying it can result in dropping some net.* traffic. Here are the steps:
Add !net.* to the end of each line in hosts.nntp (to the patterns key in incoming.conf for newer versions of INN) which does not refer to a sound site and which either has no hierarchy restrictions or includes "*". This tells INN to refuse posts to net.* from those hosts. (Note: If you're running a version of INN prior to 2.0 and you have compiled INN with REMEMBER_TRASH set to DO, the default, this will cause you to miss some net.* posts unless you install the patch mentioned below.) If you have a sufficiently new version of INN (2.2 I know has this), you can use @net.* instead and catch crossposts as well.
Add @net.* to all of your feeds, even those to sound sites, to plug leaks (requires INN 1.5 or later; if you're using 1.4 and haven't back-ported the addition to allow @ patterns, use !net.*).
Add separate feeds to your sound peers for just net.*. A typical entry would look like this:
peer-usenet2/peer.path.name:@*,net.*/4gh:Ad,Tf,Wnm:
(with obvious differences if you use innfeed or nntplink). Note the inclusion of @*; the combination of this and @net.* on your other feeds will stop you from propagating unsound crossposts between net.* and non-net.* groups in most cases. Note also that this requires a distribution of 4gh on posts in order for you to propagate them, which will also help considerably in preventing you from propagating unsound posts. If you're using INN 1.4, just use "!*,net.*" instead of "@*,net.*".
Make whatever changes you need to so that these additional feeds will be sent out. (For example, if you use nntpsend, you'll need to edit nntpsend.ctl). This is the same as when adding any other new feed.
Make sure that you're accepting the "4gh" distribution. You may have to add it to the distributions on your ME line in newsfeeds unless you're already accepting all distributions.
Add the net.* groups that you want to carry. A complete list is at http://www.usenet2.org/checkgroups.txt; you can run this through ~news/bin/control/docheckgroups if you'd like or add some more limited set.
Edit distrib.pats and add the line:
90:net.*:4gh
This forces the required distribution of "4gh" for all articles posted locally to net.*.
Decide how you want to handle control messages for net.*. If you have PGP verification installed (HIGHLY recommended) and would like to carry all net.* groups, add the following lines to your control.ctl file:
checkgroups:control@usenet2.org:net.*:verify-control@usenet2.org=mail newgroup:control@usenet2.org:net.*:verify-control@usenet2.org=newgroup rmgroup:control@usenet2.org:net.*:verify-control@usenet2.org=rmgroup
See http://www.usenet2.org/control@usenet2.org.asc for the PGP public key.
If you do not have PGP installed but would still like to honor control messages automatically (this is somewhat dangerous), use:
checkgroups:control@usenet2.org:net.*:mail newgroup:control@usenet2.org:net.*:doit=newgroup rmgroup:control@usenet2.org:net.*:mail
You should periodically verify your list of groups against the canonical checkgroups at the above URL if you do this.
Usenet II sites are not required to carry all net.* groups, but they are required to honor all valid rmgroups (we want to have an effective group removal method and not be plagued by dead weight sitting around). You can choose to ignore all control messages for net.* if you wish, but if you do this, you'll need to periodically check the canonical list of groups and remove any groups you carry that have become defunct.
Set up moderation forwarding for Usenet II newsgroups. To get submissions for moderated Usenet II groups to go to the right place, add the line:
net.*:%s@moderators.usenet2.org
to your moderators file.
OPTIONAL: For versions of INN before 2.0, patch INN to work around the behavior of REMEMBER_TRASH. Previous to 2.0, if you're running with REMEMBER_TRASH set to DO, posts rejected due to a ! entry in hosts.nntp are remembered and always rejected from then on. This causes a problem given the existence of leaks. It often happens that a sound Usenet II article leaks out and arrives at your site via an unsound path faster than a sound path. If this happens, a stock INN server configured as above will never accept the article since it will reject its first appearance due to the unsound path.
http://www.usenet2.org/inn.remembertrash.patch.txt is a patch to INN that causes posts rejected due to ! entries in hosts.nntp not to be remembered even under REMEMBER_TRASH, which neatly solves this problem. It is an extremely short patch and has been thoroughly tested with a variety of versions of INN, and I definitely recommend installing it. That being said, it's not required to run a sound site.
A version of this patch has already been integrated into INN as of version 2.0.
Congratulations! Tell your sound peers to go ahead and turn on your net.* feed; you're now a sound site.
Again, it's important to realize that being a sound Usenet II site is an ongoing commitment, and something that you'll have to think about when adding new feeds. That being said, I've been running with the above setup without any problems for quite some time, and with a little bit of care it's very easy using INN to remain sound. Just make sure that you always add @net.* to the feed entry for any new unsound peer you add, and that you always add !net.* to their hosts.nntp line. It would also be excellent if you could monitor net.config.leaks and net.config.unsound-sites on a semi-regular basis (although we'll try to contact you directly if there is any problem involving your site).
Older versions of INN didn't have any way to reject crossposts between net.* and non-net.* groups, since only ! patterns were allowed in hosts.nntp. They also had a bug in the way ! patterns were handled. The combination meant that if someone crossposted an article between alt.foo and net.foo, that article would be accepted by your server even if the hosts.nntp entry for the peer you're receiving it from has !net.*, and furthermore it will be propagated to all of your alt.* peers even if you have @net.* in their newsfeed entries. This was because !net.* just told INN to ignore net.* groups for the purposes of determining if you can accept an article (so you accept the article as a post to an alt.* group), and furthermore then masks the existence of the net.* groups in the Newsgroups header to all of the rest of INN (so @net.* never catches and you propagate the article as an alt.* post). This bug has been fixed in newer versions of INN (by at least version 2.2).
If you have separate reading and feeding servers, you can adjust your internal feeds between the servers to filter out even more unsound posts. Just make sure that both your feed from your feeding server to your reading server and your feed from your reading server to your feeding server are designed along the lines above (with Usenet II separated out with a required distribution header) and not only will you not propagate those unsound posts but they won't even show up on your reading server. You can also use split feeds from your reader server to your feeding server to prevent any crossposts between net.* and non-net.* groups from ever leaving your reading server (by putting @net.* in the main feed and @* in the Usenet II feed).
It is, of course, possible (and in many ways much easier) to implement a number of the other net.* rules using Perl filters. Perl filters are not required, but if you would like to use them to stop crossposts between net.* and non-net.* groups, implement the Rule of Three (see the Usenet II rules for details), or implement other post restrictions, that's an excellent idea. If you'd be willing to make the results available to everyone else, even better! Cleanfeed (at <http://www.mixmin.net/cleanfeed/>) has support for rejecting Usenet II rule violations. If, however, you don't have INN compiled to use Perl or don't have the time or knowledge to do this, don't worry about it, provided that you have some other infrastructure in place to handle rule violations.
| Russ Allbery > FAQs and Documentation | Soundness for C News > |