setting up syslog for distributed application logging

as you probably know, syslog is a *nix logging subsystem, typically used for system logging and kernel message trapping. you might not know that it's also an good tool for application logging, including large distributed systems. it has been around since the dawn of time and has very broad language, application and device support. here's a quick how-to to get you started, with debian etch examples.

syslog configurations vary in complexity from simple logging to local files to network based logging to a single centralized logging server (or set of logging servers).

terminology

here are some syslog terms you should be familiar with:

• facility: a syslog facility is an application space for log segmentation. examples are daemon, ftp, kern, lpr, mail. for application logging, you'll probably stick to the user defined facility space local0 through local7. unfortunately, you can't add your own facilities.
• priority: the priority tells you how important a message is, and can have the following values: debug, info, notice, warning, crit, alert, emerg

preparing your syslog server

to prepare your server for remote logging, you need to give syslogd the -r argument to tell it to accept distributed logging requests. you can do this by creating the following line in /etc/default/syslogd note: this example also turns off the arguably annoying -- MARK -- lines in the log.

next, choose a file where you'll want your application logs written to, and a facility that you'd like to use. add this configuration to /etc/syslog.conf, for example

the choice of user definied facility is somewhat arbitrary. local0 is a decent choice if you don't want to think about it.

now, restart your syslog daemon.

preparing your application server

you are now ready to configure your application server's logging. syslog allows you to easily configure which priority messages get logged locally, which get logged remotely (and much more). see: for more information. here's one example from my application server's /etc/syslog.conf file:

note that my logging server's hostname is "logserver". don't forget to restart your syslog daemon after changing this file.

making some log statements

you can test your setup easily using syslog's shell interface, logger, see: for more information.

here's a simple script that writes a couple of log statements:

note: purely as an example, i've added the date to my log statements to ensure each one is unique and avoid syslog's handy repeated line suppression

now take a look at the file /var/log/application on both your application server and your logging server. you'll see critical logs (pants on fire) in the log server, and all logs on the application server.

integration into other systems

most applications and programming have a syslog integration, and i won't bore you with the details. here are some links to illustrate the point:

• PostgreSQL
• Drupal 6.0
• php-log
• log4j

further reading

sometimes you want critical logging requests to be more "active" than a simple log statement written to a logfile. i've just written a short article describing how to get real time notification of critical syslog events by sms or email. you can find the article here.

tech blog

if you found this article useful, and you are interested in other articles on linux, drupal, scaling, performance and LAMP applications, consider subscribing to my technical blog.