iceflatline

I’ve grown tired of connecting to each host individually in my network to examine their log files. In addition to logging events locally, I would like these hosts to send their logs to a designated host in my network, resulting in a single location where I can examine and analyze all logs.

This post describes how to setup and configure a machine running FreeBSD to be a system log or “syslog” server, receiving incoming log events from other hosts in the network. A second machine, also running FreeBSD, will be configured to send its log events to the syslog server.

For purposes of example, we’ll use the hostname “server” for the machine hosting our our syslog server, and “client” for the other machine – the one sending its log events to the syslog server. All steps involved assume that FreeBSD is installed and operating correctly on both machines. All commands are issued as the root user.

The versions for the software used in this post were as follows:

• FreeBSD 11.0-RELEASE

Let’s get started…

Configure the syslog server

First, we need a file in server’s /var/log directory to host the log events coming from client. For our example, we’ll make this file name the same as client’s hostname. While you don’t need to use the .log extension, I find it helpful as it clearly indicates the purpose of the file:

Next we need to add a couple of options to syslogd, the FreeBSD utility that reads and logs messages to the system console and log files. Use sysrc to add the following line to /etc/rc.conf, substituting the IP network and network mask for your own:

The -4 (IPv4) option forces syslogd to listen for IPv4 addresses only.

The -a (allowed_peer) option specifies which clients are allowed to log to this syslog server. This option can take the form of IP address/mask:service, such as “-a 192.168.10.1/24:*” (the `*’ character permits packets sent from any UDP port), or hostname.domain, such as “-a client.home”, or “-a *.home” (assuming that the hostname can be successfully resolved to the correct IP address in the network). Multiple -a options may be also be specified. In this example, allowed_peer will the form of any host within an entire IP network, in this case 192.168.1.0/24.

Finally, the -v opton indicates verbose logging. If -v is specified once, the event’s numeric facility and priority will be added to the log. If specified more than once, the names of the event’s facility and priority (e.g., “user.notice”) are also added to the log.

Now we need to add some lines to server’s /etc/syslog.conf file, the configuration file for syslogd. First, the name of server’s hostname, preceeded by a + character, must be added to top of the file – before any existing syslog options (i.e., right before *.err; …, etc.) – so that those existing options will be applied only to log events generated by locally by server. If we did not add this line then all those options would also be applied to the log events that arrive from client. In other words, any options after a +(some_hostname) in the this file will apply until the next +(some_hostname) is parsed:

Then add following lines to the bottom of /etc/syslog.conf after the last !* , substituting the .home domain for your own:

The first line specifies that remote log events will be arriving from client. client can be specified using either its hostname or its IP address. Note that when using a hostname the domain name must be appended to it. In either case, the hostname.domain or host ip address is preceded by a + character.

The second line contains parameters to control the handling of incoming log events from client, specifically a selector field followed by an action field. The syntax of the selector field is facility.level. The facility portion describes which subsystem generated the message, such as the kernel or a daemon, while the level portion describes the severity of the event that occurred. Multiple selector fields can be used for the same action and should be separated using a semicolon (;). In our example we’ll use the * characters in the selector field to match any log events received by client.

The action field denotes where to send the log message. In our case, log events will be sent to the log file we created previously. Note that spaces are valid field separators in FreeBSD’s /etc/syslog.conf file. However, other nix-like systems still insist on using tabs as field separators. If you are sharing this file between systems, you may want to use only tabs as field separators.

Managing the log files

The file /var/log/client.log will grow over time, making it difficult to locate useful event information as well as taking up disk space. FreeBSD mitigates this problem using using newsyslog, a built-in utility that, among other things, periodically rotates and compresses log files. newsyslog is scheduled to run periodically by the system crontab (/etc/crontab). In its default configuration, it runs every hour.

newsyslog reads from its configuration file, /etc/newsyslog.conf in order to determine which actions to take. This file contains one line for each log file that newsyslog manages. Each line is comprised of various fields which control the log file’s owner and group, permissions, and when the log file should be rotated. In addition there are several optional fields for controlling log file compression and programs that should be signaled when the log file is rotated. Each field is separated with whitespace.

In order to have newsyslog recognize client’s log file, we’ll place the following line at the botton of /etc/newsyslog.conf:

In this example, the file permission for /var/log/client.log is set to 640. newsyslog will retain up to five archive files, and rotate the file when its size reaches 100 kB. The * character in the when column instructs newsyslog to ignore a time interval, a specific time, or both and instead only consider the size of the file when determining whether or not to rotate the file. The J flag tells newsyslog compress the rotated log file using bzip2, and the C flag tells newsyslog to create the log file if it does not already exist.

Finally, let’s restart syslogd and newsyslog on server:

Configure the client

Let’s move on now and configure client so that it will send its event logs to server. Open client’s /etc/syslog.conf file and add the following line after the last !*, to instruct client to send log events of any facility and level to server:

server can be specified using either its hostname, hostname.domain or its IP address, preceded by a @ character.

Now let’s restart syslogd on client:

Finally, let’s make sure client is sending its log events to server using the logger utility. Logon to client and issue the follow command:

Now login to server and and check client’s log file:

You should see the message you sent using the logger utility:

Conclusion

That’s it. In addition to logging events locally, the client host will send its logs to our syslog server, resulting in a single location where log events can be examined and analyzed.

References

NEWSYSLOG(8)
NEWSYSLOG.CONF(5)
SYSLOGD(8)
SYSLOG.CONF(5)