Syslog Central Logging Server

In 2022, IS and SE IE worked together to set up a test syslog server to see if centralized logging could be achieved with this existing technology. The idea was to centralize logs for various applications so that we could share in supporting other engineering applications and also point monitoring and troubleshooting tools to those logs.

Set Up

IS created an ubuntu server (syslog-server.shore.mbari.org) and installed the docker daemon. In order to make the logs available in an easy to access format, we attached a share from Titan for the server to write the logs to. This took some setup to get user permissions correct for this to work.

First, the titan share was created and Neil made it only visible by direct mounting so you have to know the share is there as it is not visible to the world (for security reasons). This share was then mounted to /data/devops on that machine. A local account was created with the UID/GID of 11906:11906 that aligned with the UID of the AD account that has permissions on that share. Now, in order for the UID to carry through, but to not have to do anything special in Docker, peter configured the docker daemon by editing the /etc/docker/daemon.json file to look like:

    {
      "live-restore": true,
      "group": "docker",
      "userns-remap": "docker_user"
    }

This tells the docker daemon to treat all interactions with the host from all containers as if they were being generated by the user "docker_user". This allows the containers to still run like they normally would (using root inside the container) and then doing the user request mapping over to the docker_user account when it tries to interact with the host. This essentially allows the root account inside the container access the titan share as the user "docker_user". Very cool!

For the syslog server, I decided to try the syslog-ng docker image. I created two files in the /opt/syslog-ng directory (docker-compose.yml and syslog-ng.conf). Here is the docker-compose.yml contents:

    version: "3"
    services:
    syslog-server:
        image: balabit/syslog-ng
        restart: unless-stopped
        ports:
        - "514:514"
        volumes:
        - /data/devops:/var/log/syslog
        - ./syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf

And the contents of the syslog-ng.conf file is:

    @version: 3.35

    template t_network {
      template("${MESSAGE}\n");
    };

    source s_network {
      network();
    };
      destination d_network {
      file("/var/log/syslog/${HOST}/${PROGRAM}/${YEAR}-${MONTH}-${DAY}.log" template(t_network) create-dirs(yes) dir-perm(0777) perm(0777));
    };

    log {
      source(s_network);
      destination(d_network);
    };

    source s_internal {
      internal();
    };

    destination d_internal {
      file("/var/log/syslog/messages" perm(0777));
    };

    log {
      source(s_internal);
      destination(d_internal);
    };

The docker-compose.yml file mounts the titan share to the container as well as maps in the syslog-ng.conf file into the container to configure the syslog-ng server to write the files the way we want it to. Basically I set it up so that it just writes the message that comes in as-is to the file and then organizes the log files in the /var/log/syslog directory by the host and container name and then divides the files up by year, month and day. It also sets the permissions to wide open (although I don't think that is necessary with the titan share). I started the docker container using docker compose and it was off to the races!

Viewing the Logs

The easiest way to view the logs is to connect to the Titan share DevOps. It is not available through browsing (for security reasons) so you have to know the mounting point and mount it directly. The mount point is:

    smb://titan.shore.mbari.org/DevOps

In this share, you will see the logs organized by hostname (or IP address) and then by the name of the container that is generating the logs.

Configuring Docker to Use Syslog

In order for you to use the central syslog server, you need to configure the docker daemon to use syslog as the logging driver and point the it to the syslog server. As an example, here is the /etc/docker/daemon.json file from the telepresence server (telepresence.mbari.org). Note that if your server is in the DMZ, you will need to have IS put a firewall hole so that your server can get to port 514 on syslog-server.shore.mbari.org.

    {
      "log-driver": "syslog",
      "log-opts": {
        "tag": "{{.Name}}",
        "syslog-address": "tcp://syslog-server.shore.mbari.org:514"
      }
    }

You will need to restart the docker daemon and if you have containers running already, the will need to be deleted and rebuilt/restarted to get the logs to start flowing to the syslog server. Also note that docker logs will still work if you want to view the console logs from the containers.