Knowledge Required: Comfortable in a Linux shell

Tools required: Linux derivative (this example uses Debian)

Syslog is a common transport for sending application or security logs over a network. It’s more than likely that you want to aggregate all these syslog feeds into a central location so you can manage them effectively. Commonly this is done for security operations so that these feeds can be passed onto a platform such as a SIEM to identify suspicious behaviors within logs.

This guide will take you through how to setup a syslog server using rsyslog, control where logs are stored and how you can send them onto another syslog toolset such as a SIEM.

Configure rsyslog

Install rsyslog

To start with, let’s create create the configuration for the syslog server. We’re going to use the rsyslog package which can be installed using the package manager shipped with your Linux distro. In my case, it was pre-installed but could have been installed on Debian with sudo apt update -y && sudo apt install rsyslog

With rsyslog installed you should have a file called rsyslog.conf in the /etc (everything-to-configure) folder. You’ll also have a folder with the path /etc/rsyslog.d. Lots of guides may direct you to put your server config directly in /etc/rsyslog.conf but having all your configuration in one file can lead to it getting messy and you can end up having conflicting configurations. Instead, we’re going to create a new file in /etc/rsyslog.d/.

Within this directory, files are processed in order of the leading number at the beginning of the configuration filename. For example, take a look at the following:

nathan@syslogsec01:/etc/rsyslog.d$ ls -alh
total 20K
drwxr-xr-x  2 root     root     4.0K Aug 21 15:52 .
drwxr-xr-x 85 root     root     4.0K Aug 14 10:33 ..
-rw-r--r--  1 root     root       89 Aug 13 16:02 10-exclusion-process.conf
-rw-r--r--  1 omsagent omiusers  187 Aug 13 15:49 95-omsagent.conf
-rw-r--r--  1 root     root      748 Aug 21 15:52 96-rsyslog-server.conf

The files will be processed by rsyslog in the list they are presented, starting from the lowest number to the highest. This is an important concept if you’re working on a server that has other config files. In the above example, our syslog server configuration is stored in the file with the highest value, meaning any filters applied in 10-exclusion-process would be applied first. Having an exclusions file is not essential at this stage and may serve as content for a future blogpost!

rsyslog server configuration

For now, let’s create a configuration file in /etc/rsyslog.d/ with the name 96-rsyslog-server.conf. Set the following to be the content of this file:

# setup an rsyslog server in a custom file so we can manage it easily 

# make a remote ruleset for files
# set a remote logging template
$template RemoteLogs,"/var/log/remotesyslogs/%HOSTNAME%/%SYSLOGFACILITY-TEXT%.log"

# Remote Logging
$RuleSet remote
*.* ?RemoteLogs
# Send messages we receive to another syslog server
# *.* @@192.168.0.12:514 

# enable 514 UDP
$ModLoad imudp
$InputUDPServerBindRuleset remote
$UDPServerRun 514

# enable 514 TCP
$ModLoad imtcp
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514

The above config does the following:

  • Sets a syslog template called RemoteLogs
    • The RemoteLogs template will store all remote logs in /var/log/remotesyslogs and all received logs will be indexed by a hostname and then the syslog facility they were sent with
  • We load the module for listening on syslog on TCP & UDP both on port 514 as some platforms can only send syslog via UDP

Once you’ve copied this into 96-rsyslog-server.conf you will need to restart rsyslog on the host for the configuration to take effect. This can be done with sudo systemctl restart rsyslog

If you have another syslog platform (such as a SIEM) and you want to send any received logs from the syslog server to an upstream server then you can uncomment # *.* @@192.168.0.12:514, setting a source IP and port of the upstream server. Note that in rsyslog @ is used to send logs via UDP and @@ is used to denote TCP. In this example, it will send all logs of facility and severity which is denoted. *.*. The example line from our configuration would send to a server at 192.168.0.14 on port 514 using TCP

Setting up other linux servers to syslog to this destination

Now we’ve already seen the rsyslog syntax for how to configure logging to a remote server, we can use this on one of our target endpoints that we will send syslog from. Again, this will need the rsyslog package installed.

On our target endpoint edit /etc/rsyslog.conf and add the following line at the bottom:

*.* @@<YOUR SYSLOG SERVER>:514

Note: We are using TCP because TCP is more reliable and actually will ensure packet ordering on the network level. TCP can be used for reliability but does incur some processing overhead on the client and server. UDP can be used if essential

Again, after saving the file, you will need to apply the rsyslog with sudo systemctl restart rsyslog

Validating the configuration

If you wish to validate rsyslog is working on the client and server, you can run sudo systemctl status rsyslog and confirm the state is active. If it isn’t, then you probably have mistyped something and should retrace your steps before continuing.

On our syslog server, we should now be able to see logs from the client we just setup. In this case, I can see the configuration has been successful on a number of hosts:

nathan@syslogsec01:/var/log/remotesyslogs$ ls -alh
total 24K
drwxr-xr-x 6 root root 4.0K Aug 25 20:20 .
drwxr-xr-x 9 root root 4.0K Aug 21 00:00 ..
drwxr-xr-x 2 root root 4.0K Aug 21 16:25 192.168.0.12
drwxr-xr-x 2 root root 4.0K Aug 14 11:00 ansible01
drwxr-xr-x 2 root root 4.0K Aug 14 13:00 nessus01-v2
drwxr-xr-x 2 root root 4.0K Aug 14 10:50 zeus.Home

If we start investigating the content of those directories, we can see that each syslog ‘facility’ has been broken down into a file. Facilities are used within syslog to categorise the data in the log being sent from the client. For example, we can guess from the below that logs relating to authentication are going to be in the auth file:

nathan@syslogsec01:/var/log/remotesyslogs/ansible01$ ls -alh
total 5.0M
drwxr-xr-x 2 root root 4.0K Aug 14 11:00 .
drwxr-xr-x 6 root root 4.0K Aug 25 20:20 ..
-rw-r----- 1 root adm  322K Aug 25 20:01 auth.log
-rw-r----- 1 root adm  1.7M Aug 25 20:17 authpriv.log
-rw-r----- 1 root adm   66K Aug 25 20:17 cron.log
-rw-r----- 1 root adm  2.3M Aug 25 20:14 daemon.log
-rw-r----- 1 root adm  2.7K Aug 21 16:00 syslog.log
-rw-r----- 1 root adm  672K Aug 25 20:00 user.log

The line which controls this behaviour is in our syslog server configuration and is: $template RemoteLogs,"/var/log/remotesyslogs/%HOSTNAME%/%SYSLOGFACILITY-TEXT%.log".

Keep it tidy!

Having a centralised syslog server means that your log files are probably going to grow in size pretty quickly. Thankfully, Linux has an answer for that called logrotate. Logrotate will allow us to compress old syslog files on disk or compress them when they’ve reached a certain size. To do this, we’re going to add a custom logrotate config. If you haven’t got logrotate on your system, you can install it on Debian with sudo apt update -y && sudo apt install logrotate

Similar to rsyslog, there is a /etc/logrotate.conf file and a /etc/logrotate.d folder. We’re going to make a custom config file in the logrotate.d folder to keep it nice and tidy. As we know all our remote syslog files are stored in /var/log/remotesyslogs we can apply log rotate to all files contained within that directory.

Create a new file with the path /etc/logroate.d/rsyslog-server-rotate.conf and add the following content:

/var/log/remotesyslogs/* {
    daily
    rotate 3
    size 10M
    compress
    delaycompress
}

Once copied, restart the logrotate service with sudo systemctl restart logrotate

What this configuration does can be broken down to:

  • Every day log rotate will check if any of the logs are larger than 10M
  • If they are, it will compress and save a copy of the log making sure the most recent log does not get compressed
  • Will keep 3 copies of the previously compressed logs

If disk space is an issue these parameters can be tuned to account for this.

Conclusion

Now you’ve got a basic working example of an rsyslog server! We haven’t touched more complex configurations (such as encrypted syslog) but this should be enough to get you up and running! Remember that you should be limiting who can send logs into your syslog server for security purposes and so that you have tighter controls on disk usage. This can be done by using iptables on the host itself or a dedicated network firewall.

EOF break