| < lsmounts | Russ Allbery > Software | AFS Mount Point Tracking > |
A distributed system is one that stops you from getting any work done when a machine you've never even heard of crashes.
— Leslie Lamport
This is a collection of Nagios plugin scripts for monitoring various aspects of AFS servers. They all follow the Nagios standards for check_* scripts (although don't support a few features) and use exit statuses that give Nagios the right information. They don't use any Nagios-specific libraries, however, and should be suitable for use with other monitoring systems such as mon. Any monitoring system that understands remote checks for services should be able to use these scripts with some adaptation.
check_afsspace uses vos partinfo to check the available space on each partition on a file server. It reports a critical error if the percentage used is above a configurable threshold (90% by default) and a warning if it is above a lower configurable threshold (85% by default).
check_bos runs bos status on a file server or volume location server and scans the output, making sure that all commands are running normally and the file server isn't salvaging. If it sees any output it doesn't expect from bos status, it reports that output in an alert.
check_rxdebug runs rxdebug against a file server and looks for any client connections that are in the state "waiting for process." This indicates client connections that are blocked waiting for a file server thread. We've found this to be a reliable test for detecting serious file server performance problems. It reports a critical error if the count of such connections is above a configurable level (8 by default) and a warning if it is above a lower configurable threshold (2 by default).
check_udebug runs udebug against a ubik service (vlserver, ptserver, kaserver, or buserver) and makes sure that it is in a reasonable state. It checked to be sure that there is a sync site for the service, and when there is, that the sync site believes that the recovery state is 1f (indicating that all of the slaves have the same version of the database).
These scripts are primarily intended to be run from inside Nagios, but can also be run separately via any other monitoring system or even manually. They are all written in Perl and require various versions, but 5.005 is sufficient to run all of them.
They use the standard AFS utilities to do their work, so you need to have rxdebug, udebug, bos, and vos available. Obviously, since these scripts monitor AFS, you want to have copies of these utilities on local disk rather than relying on copies out of AFS. You may want to change the logic at the top of the scripts to search for a suitable version.
The scripts all have default timeouts and check_afsspace and check_rxdebug have default thresholds that you may want to change. You may also want to look at the regexes for acceptable bos status lines (for example, if you want to get paged whenever your file server has a core file, you will want to modify the regex to not filter that out).
The programs:
| check_afsspace 1.16 | 2006-03-17 | Download |
| check_bos 1.7 | 2006-03-17 | Download |
| check_rxdebug 1.11 | 2006-03-17 | Download |
| check_udebug 1.3 | 2006-03-17 | Download |
Documentation:
Copyright 2003, 2004, 2005 Board of Trustees, Leland Stanford Jr. University.
These programs are free software; you may redistribute them and/or modify them under the same terms as Perl itself. This means that you may choose between the two licenses that Perl is released under: the GNU GPL and the Artistic License. Please see your Perl distribution for the details and copies of the licenses.
| < lsmounts | Russ Allbery > Software | AFS Mount Point Tracking > |