| Russ Allbery > Software > Orphaned Software > filter-syslog | filter-syslog Changes > |
(Filters a syslog file and mails the results)
filter-syslog [-hnov] [config] < syslog
filter-syslog parses a log generated by syslog, filtering out all of the boring lines as configured in config, and then mails the remaining lines to the address specified in config. It expects the log file on standard input, and is designed to run from an analyze action in newsyslog(8), although it can be used in other situations as well. It can also parse Apache error logs and do similar filtering.
If config isn't an absolute path, it's taken to be relative to either /etc or /etc/leland, wherever the file is found (searched in that order). If config is not specified, it defaults to filter-syslog.conf and is looked for in both /etc and /etc/leland.
Lines containing -- MARK -- and sysklogd restart messages on Linux,
which look like:
Sep 10 04:02:07 example syslogd 1.4.1: restart.
Apr 1 23:55:01 example syslogd 1.4.1#10: restart.
Apr 1 23:55:01 example syslogd 1.4.1#10: restart (remote reception).
Apr 1 23:55:09 example exiting on signal 15
are always ignored.
Messages of the form:
Apr 28 07:09:40 10.1.1.1 Message forwarded from example.org: \
program[36398]: some log message
(line split only for readability in this example) will be parsed exactly as if they had said:
Apr 28 07:09:40 example.org program[36398]: some log message
This format is used by OpenBSD for forwarded syslog messages.
Please note that this is not intended to be a security tool or a real-time monitoring tool, but rather a tool to make sure that system administrators are aware of unusual log messages that might indicate server problems or failing hardware. An intrusion detection system would work differently and would be more paranoid, and a real-time monitoring tool wouldn't run in batch mode. There are other tools available to do that type of monitoring.
Print out this documentation (which is done simply by feeding the script
to perldoc -t).
Rather than sending the results via e-mail, instead print out the non-boring lines that would have been sent via e-mail to standard output. Useful for testing filter rules.
Display the hostname field (from the input syslog) in the output.
Print the version of filter-syslog and exit.
There are three types of valid lines in the configuration file; variable settings, filter patterns, and includes of other configuration files. A variable setting looks like:
variable = value
where value can contain whitespace (but can't begin with whitespace). A filter pattern looks like one of:
program: /regex/
/program/: /regex/
/regex/
/regex/ ... /regex/
where program is the name of a particular program (the filter line
will only apply to log entries from that program) and regex is a Perl
regular expression matching lines that are "boring" and shouldn't be
reported. Any trailing whitespace in the syslog line will be removed
before matching it against the regex. Slashes (/) in regex do not
have to be escaped. Each of these lines must be all on one line.
When a line is in Apache error log format, the program for that line will
be set to apache-level where level is the log level for that line
(notice, warn, etc.).
If program is surrounded by slashes (/), it is a regex and any
program name that matches that regex will have that filter line applied.
If program is * regex will be applied to all lines, regardless of
what program they're from.
If program is not present, as in the last two forms, the regex is matched against the entire syslog line, including the timestamp, and the line will be ignored if the regex matches. This can be used to match logs in a non-standard format, such as ones without a program name or with a program name containing spaces.
If the line contains two regexes separated by ..., this indicates a
range of lines. All lines between a line matching the first regex and a
line matching the second regex will be ignored, including the matching
lines. Both regular expressions are matched against the entire line,
including the timestamp and program. There must be no more than 1000
lines in the range; if more than 1000 lines are encountered after the
start regex, filter-syslog will stop looking for the end regex and then
parse all the lines normally.
Finally, a line like:
include /path/to/file
includes another configuration file at /path/to/file. The path can be a directory instead of a file, in which case every file in that directory that does not begin with a period is included (in no defined order).
The following variables are recognized:
The address to which to mail the filtering results. No mail will be sent if all of the input lines are filtered out by the regexes provided. This variable must be set and may not contain any backslashes or single quotes.
The address from which to mail the filtering results (used for the envelope sender and the To: header). If not set, no address will be given to sendmail, which will result in the mail system picking some default value based on the user filter-syslog is running as. The value of this variable may not contain any backslashes or single quotes.
The value to use for the Subject: header of the filtering results. If you
include $h in the value, it will be replaced with the hostname. This
variable must be set.
If there are any input lines that don't match one of the filter rules, they will be mailed to the value of alert with a subject given by subject.
Filter /var/log/syslog using /etc/syslog.filter as a configuration file.
filter-syslog syslog.filter < /var/log/syslog
Here's a sample configuration file that filters out normal Kerberos
messages and sends the result to root@example.com with a Subject: header
of example syslog filter results:
alert = root@example.com
subject = example syslog filter results
kftgtd: /^connect from /
klogind: /^connect from /
kshd: /^Executing .* for principal /
kshd: /^Shell process completed\.$/
kshd: /^connect from /
Instead of the three separate lines to filter out TCP wrappers messages, one could instead use the line:
*: /^connect from /
to filter out all syslog lines that begin with connect from, but this
runs a larger risk of filtering out messages that would be of interest.
The rule:
apache-warn: /^FastCGI: /
would filter out all Apache error log messages about FastCGI. The rule:
apache-debug: /.*/
would filter out everything in an Apache error.log logged at debug level.
The filter pattern:
/^\w{3} [ :0-9]{11} \S+ \w+\[\d+\]: connect from /
would match any syslog line from any program beginning with connect
from. This regular expression is matched against the entire line; notice
that the timestamp and host identifier have to be matched, as well as the
PID in brackets after the program name. In this specific case, there is
no reason to use such a rule since filter-syslog can parse that line
into a program name and message, but this sort of rule can be used to
match any arbitrary syslog line that filter-syslog may not otherwise be
able to parse.
Finally, the configuration line:
/START/ ... /END/
would filter out every log line between a line containing START to a
line containing END, inclusive. (This example isn't particularly
useful, but the regular expressions can of course be more complex.)
If the configuration file given on the command line isn't an absolute path, it is looked for first in /etc and then in /etc/leland. This default can be changed by editing the beginning of this program.
The default configuration file, if none is given. The paths will be searched in the above order.
The rule that ignores -- MARK -- lines, which are automatically
generated by (at least) Solaris syslogd at periodic intervals if
requested, could be exploited to hide messages from filter-syslog that
an administrator may want to see. Please again note that this is not a
security tool. However, a better regex should be developed and used
instead, regardless.
There is no protection against inclusion loops (a configuration file that includes another file which then includes the first file).
As of version 1.20, filter-syslog removes trailing whitespace from syslog lines before seeing if the lines match the provided regexes. Earlier versions did not do this. You may need to change your regexes when upgrading from 1.19 to 1.20.
newsyslog(8)
The current version of this program is available from its web page at <http://www.eyrie.org/~eagle/software/filter-syslog/>.
Russ Allbery <rra@stanford.edu>. Patch for --hostname from Steve Benson.
Copyright 2002, 2003, 2004, 2006, 2007, 2009, 2010, 2011, 2012 The Board of Trustees of the Leland Stanford Junior University.
This program is free software; you may redistribute it and/or modify it under the same terms as Perl itself.
| Russ Allbery > Software > Orphaned Software > filter-syslog | filter-syslog Changes > |