| < newsyslog README | Russ Allbery > Software > Orphaned Software > newsyslog | newsyslog Changes > |
(Log file rotation and periodic maintenance)
newsyslog [-bchv] [-s minutes] [config]
newsyslog performs periodic maintenance of log files. The most common action is rotating log files and archiving the old logs to one or more places, but it can also periodically copy logs to a central repository without truncating them. It can run commands both before and after log rotation to handle stopping, restarting, or sending signals to servers, and it automatically compresses logs when it archives them.
To support automated log analysis, newsyslog can run a program on each log file as it's archived, and can also run a filter command on the log and only archive the output of the filter (allowing uninteresting information to be discarded before archiving).
What newsyslog does when it runs is entirely controlled by its configuration file. The syntax of configuration file is described below. The path to the configuration file to use can be given on the command line, or if not given will default to @sysconfdir@/newsyslog.conf.
Use bzip2 to compress log files rather than the default (gzip). This option is only available if libbz2 was available at compile time. If libbz2 was available and libz was not, this option may be required.
Load the configuration file and check its syntax, but don't do anything. If the configuration file syntax is valid, a message is printed and newsyslog exits.
Prints a usage summary and exits.
Normally, all dates and times included in file names written by newsyslog are based on the current time: the time at which newsyslog runs to rotate logs. However, it can be useful to have one's logs cover roughly a calendar day and archived with a date to match that day, which means running newsyslog via a nightly cron job just before midnight. This normally works, but if anything delays the newsyslog run slightly (such as slow cron jobs ahead of it in the /etc/cron.daily sequence), it will then archive logs with dates for the next day. This may then cause file name conflicts and other issues.
The -s option can be used to avoid this. When given, all timestamps
are skewed backwards by minutes. So, for example, newsyslog -s 60
will archive logs with file names as if newsyslog had been run one hour
earlier than it actually was. This can be used to tolerate some delays in
running newsyslog and still name the logs after the day primarily
covered by the log file.
Note that if this option is used and rotation of a log fails, the saved temporary log file that will be picked up by the next run of newsyslog will already have its timestamp skewed. So if you run newsyslog with this option, a log rotation fails, and you run it again, the failed log rotation will be retried with the timestamp skew configuration of the first run, not the second.
Print the version number and exits.
A newsyslog configuration file consists of any number of log classes, interspersed with variable settings. White space is generally not significant with the exception of newlines, which end various directives. Any line beginning with # anywhere in the file is considered a comment and ignored.
The syntax of a variable setting is:
set <variable> = <value>
where <variable> must start with a letter and consist solely of letters,
numbers, and underscores. If <value> contains spaces, it must be enclosed
in double quotes. (There is no way to escape a double quote at present.)
Everywhere in the configuration file where newsyslog expects a path, it
can take a variable instead. Any path which does not begin with / is
assumed to be a variable.
The other basic syntax element is a log class. A log class is a set of logs along with (optional) commands to stop and restart a daemon so that the logs can be rotated. It takes the following form:
<blockname> {
stop: (run <command> | hup <file>)
restart: (run <command> | hup <file>)
log: <name> <location> <owner> <group> <perms>
archive: <name> <archive> <count>
copy: <name> <archive>
truncate: <name>
analyze: <name> <command>
filter: <name> <command>
}
The directives must appear in this order, except that archive, copy, filter, and analyze directives may be freely mixed. <blockname> is not used by newsyslog and is essentially a comment.
The stop directive runs before logs are rotated; the restart directive runs immediately after they have been rotated (moved aside) but before they're compressed and copied into an archive location. These directives take either a <command> that will be run or a <file>. If given a file, the file is open and read and its contents are interpreted as a PID, which is sent a SIGHUP. Either argument undergoes %-interpolation (see below).
Any number of log lines are allowed. Each one specifies a log that newsyslog is going to be doing something with it. The <name> associated with the log is used to resolve %n escapes in <location> (see below) and is used to associate archive or copy directives with this log. The full path to the log is given in <location>. If there is one or more archive directives for this log, it will be rotated by moving it to a temporary file name between the time the stop and restart commands (or SIGHUPs), if any. A new, empty file will be created in its place, with the given <owner> and <group> and with permissions given by <perms> (which will be interpreted as an octal number). The <owner> and <group> can be either numeric or symbolic.
Any number of archive, copy, truncate, and analyze directives are allowed, although truncate is not allowed in combination with archive since archived logs are always replaced with a new, zero-length file. Each one must be associated, via <name>, with a log declared in a preceding log directive.
Only a single filter directive is allowed for a given log. Like an analyze directive, it must be associated, via <name>, with a log declared in a preceding log directive.
An analyze directive runs <command> and pipes the log into it. Any output from the command will go to wherever the standard output from newsyslog will go. (For example, if newsyslog is being run from cron, the output will go into mail to the user whose crontab newsyslog is running out of.)
A filter directive runs <command> on the log and copies or archives the output of the filter command rather than the raw log.
For archive and copy directives, <archive> is the full path to where the
log should be archived. Logs will automatically be compressed when
archived (and .gz, or .bz2 if -b was given, will be added to the
end of the file name). A copy directive will just store a copy of the log
in the given location, and if only copy directives are specified for a
given log, it won't be rotated. An archive directive triggers log
rotation, and takes an additional <count> parameter, which specifies how
many old logs to store in the given <archive> location. If count is 0,
just one log is stored under the given name removing any existing file
with that name. If count is greater than 0, <count> numbered logs are
stored, and each time a log is archived in that location, the highest
numbered log is deleted if there are already <count> logs there and all
the other older logs are renamed, incrementing the numbers in their names.
The current log is then archived as <archive> with .0.gz or .0.bz2
appended.
A truncate directive says to reduce the log file to zero length in place without moving the file aside and creating a new one. It can be used in combination with copy to copy the current log file and then truncate it, although be aware that if the log is actively being written to, some lines may be missing from the copy and then will be discarded by the truncate.
<command>, <file>, <name>, <location>, and <archive> all undergo %-expansion. The following special sequences are recognized:
%% The character `%'.
%d The current date in the form yyyy-mm-dd.
%D The current day of the month in the form dd.
%M The current month of the year in the form mm.
%Y The current year in the form yyyy.
%t The current time in seconds since epoch.
%m The machine hostname.
%n The name of the log being rotated.
Each time newsyslog is run, all log rotations described in the configuration file are performed. newsyslog should probably be run from cron in the standard installation to control the timing of log rotations.
Here's an example configuration file to rotate AFS authentication server log files. This includes two syslog logs and an authentication server log that requires stopping and restarting the server in order to rotate.
# The logs are archived in a directory named for the current date
# in a file named <machine>.<log> where <machine> is the hostname
# of the machine and <log> is the name of the log. Copies are also
# saved on local disk.
set authlog = /afs/ir/service/afs/logs/%M/%d/%m.%n
set syslog = /afs/ir/service/afs/logs/%M/%d/%m.%n
set oldlogs = /var/log/OLD/%n
# Save the five most recent copies on local disk.
syslog {
restart: hup /etc/syslog.pid
# name location owner group permissions
# ------ --------------- ----- ----- -----------
log: syslog /var/log/syslog root 0 644
log: auth /var/log/auth root 0 644
# name archive count
# ------ ------- -----
archive: syslog syslog 0
archive: syslog oldlogs 5
archive: auth authlog 0
archive: auth oldlogs 5
# Run syslog through a lot analysis program.
analyze: syslog /usr/local/sbin/analyze-syslog
}
afs {
stop: run /usr/afs/bin/bos shutdown %m kaserver -localauth -wait
restart: run /usr/afs/bin/bos restart %m kaserver -localauth
# name location owner group permissions
# ------- --------------------- ----- ----- -----------
log: AuthLog /usr/afs/logs/AuthLog root 0 644
# name archive count
# ------- ------- -----
archive: AuthLog authlog 0
# Run the AuthLog through a filter first to discard boring
# log entries.
filter: AuthLog /usr/local/sbin/filter-authlog
}
Note that the stop and restart commands in the afs class make use of %-expansion to get the local machine name, which is one of the arguments to the command to stop and restart the server.
The default configuration file.
Error messages are printed to stderr. In the case of errors, newsyslog tries to continue anyway if it makes sense to, doing as much as it can. (The exception is parsing errors in the configuration file, which will cause it to abort without doing anything at all.) If a log is successfully rotated but then cannot be archived, it will be left behind with a temporary file name (the normal name of the log with a period and a timestamp appended), and the next time newsyslog runs with the same configuration, it will try to archive the log again.
The current version of this program is available from its web page at <http://www.eyrie.org/~eagle/software/newsyslog/>.
Russ Allbery <eagle@eyrie.org>
Copyright 1997, 1998, 1999, 2000, 2002, 2004, 2005, 2011 The Board of Trustees of the Leland Stanford Junior University.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
| < newsyslog README | Russ Allbery > Software > Orphaned Software > newsyslog | newsyslog Changes > |