Systemd Service Failure Notification System

I run a small root server with my website, a Nextcloud instance and some other stuff for my personal use. This server runs NixOS and I deploy new versions using NixOps, a NixOS deployment tool. This setup runs smoothly and is easy and time-efficient to maintain.

However, sometimes things do break, and even the rollback features of NixOS do not help if you notice the error too late. As I am mostly the only one using the services provided by the server, I am also the only one affected. Still, I want to know as early as possible when something is broken. This allows me to plan the fix instead of fixing it in a hurry when I need to use a service.

I looked into various monitoring tools. However, most of them are complex tools designed for complex systems, not for a single server with a single regular user. Therefore, I decided to build my own monitoring tool by leveraging the power of Nix.

The idea is simple. Everything on my server that is worth monitoring runs as a systemd service. All I need to do is send a notification once a service fails, and I need to do this for every service.

I do this by creating a new systemd service called email@. This service does nothing but send a notification with the information of the associated service. It also must not trigger itself when it fails, as this could lead to an endless loop.

In case you are confused by the @ in the service name: it allows you to pass instance parameters into the systemd unit. These instance parameters can be referred to using %i. If you want to read more about it, take a look at the systemd.unit man page.

In my configuration, this service is defined as

config = {
  systemd.services."email@" = {
    description = "Sends a status mail via sendmail on service failures.";
    onFailure = mkForce [ ];
    serviceConfig = {
      ExecStart = "${sendmail} ${config.systemd.email-notify.mailTo} %i";
      Type = "oneshot";
    };
  };
};

where sendmail is defined as

sendmail = pkgs.writeScript "sendmail" ''
  #!/bin/sh

  ${pkgs.system-sendmail}/bin/sendmail -t <<ERRMAIL
  To: $1
  From: ${config.systemd.email-notify.mailFrom}
  Subject: Status of service $2
  Content-Transfer-Encoding: 8bit
  Content-Type: text/plain; charset=UTF-8

  $(systemctl status --full "$2")
  ERRMAIL
'';

This should be pretty straightforward. The service is a oneshot service that just executes the sendmail script. Setting onFailure to an empty list with mkForce makes sure that nothing is triggered when the service itself fails.

Now we need the other part as well: attaching the email@ service to every systemd service, or at least to every systemd service that is managed by NixOS.

This is done with the following code snippet:

{
  systemd.services = mkOption {
    type = with types; attrsOf (
      submodule {
        config.onFailure = [ "email@%n.service" ];
      }
    );
  };
}

Here, we append email@%n.service to the onFailure attribute of every service.

I have been using this setup for almost three years now, and I am really happy with it. There is potential for improvement (channels other than email, stopping notifications after many consecutive failures of a service, …), but so far this is all I need, and it has never let me down.

You can find the full source code of the notification system on Codeberg. If you are interested in what my server configuration looks like, take a look at my config repository.

PS: In addition to this monitoring at the systemd service level, I also use UptimeRobot to make sure all public-facing services are reachable via the internet.