How to monitor systemd timers so a failed job doesn't go unnoticed

systemd timers fail quietly: no mail, no alert. How to see failures with systemctl and journalctl, and how to add a heartbeat ping so you hear about missed runs.

· 7 min read · By the Spot Downtime team

systemd timers are a solid replacement for cron: they log to the journal, can catch up on runs missed while the machine was off, and have proper dependency handling. They share cron's biggest weakness, though. When a job fails or never runs, nobody is told. There is no MAILTO equivalent by default, and a timer that was never enabled looks identical to one that is working.

This post covers how timers and services relate, how to inspect failures, and three ways to get an alert when a timer-driven job breaks.

Timer and service: two units, two ways to fail

A timer is a pair of units. backup.timer decides when, and backup.service does the work. Failures show up in different places:

  • The service fails (non-zero exit, timeout, crash). The service enters the failed state and the journal holds its output. The timer keeps scheduling later runs.
  • The timer isn't active. It was never enabled, or was stopped, so nothing ever starts. There is no failure to see, which is why this one is the easiest to miss.
  • The run is skipped or late. The machine was off, or Persistent= isn't set so a missed run is not caught up.

A timer worth monitoring

ini
# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup

[Service]
Type=oneshot
ExecStart=/opt/backup.sh
TimeoutStartSec=2h

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup nightly

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=5min

[Install]
WantedBy=timers.target
bash
sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

Enable the timer, not the service. Persistent=true makes systemd run the job at next boot if a scheduled time was missed while the machine was down. TimeoutStartSec stops a hung job from blocking later runs forever.

Inspect what happened

bash
# When will each timer run next, and when did it last run?
systemctl list-timers --all

# Status of the timer and the job
systemctl status backup.timer
systemctl status backup.service

# Output of recent runs
journalctl -u backup.service --since yesterday

# Any failed units on the machine
systemctl --failed

In list-timers, an empty NEXT column means the timer has no future trigger. A LAST time much older than expected means runs are being missed. To verify a calendar expression before relying on it:

bash
systemd-analyze calendar "*-*-* 02:00:00"

It prints the normalized form and the next elapse time, which catches typos such as a wrong day-of-week syntax.

Common reasons a service fails under a timer but works by hand

  • Minimal environment: the unit does not load your shell profile. Use absolute paths and set variables with Environment= or EnvironmentFile=.
  • Different user: the service runs as root unless you set User=. File permissions and home directories differ.
  • Working directory: set WorkingDirectory= if the script uses relative paths.
  • Same causes as cron; see cron job works manually but fails in cron.

Option 1: get alerted when the service fails with OnFailure

systemd can start another unit when a service fails. That unit can send a message, call a webhook or mail you.

ini
# /etc/systemd/system/backup.service (add under [Unit])
OnFailure=notify-failure@%n.service

# /etc/systemd/system/[email protected]
[Unit]
Description=Failure notification for %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure.sh %i

This catches the case where the job runs and fails. It cannot catch a timer that never fired, a machine that is off, or a notification script that is itself broken, because the alert comes from the same machine as the problem.

Option 2: a heartbeat that checks in after each success

The reliable approach is the one used for cron: have the job report success to an external service that alerts when a report doesn't arrive on time. Silence becomes the signal, so it works whether the job failed, never started or the whole server is down. See how to monitor cron jobs for the general idea.

ini
[Service]
Type=oneshot
ExecStart=/opt/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 https://spotdowntime.com/api/v1/ping/YOUR_TOKEN
ExecStopPost=/bin/sh -c '[ "$SERVICE_RESULT" = "success" ] || /usr/bin/curl -fsS -m 10 https://spotdowntime.com/api/v1/ping/YOUR_TOKEN/fail'

ExecStartPost= runs only after ExecStart exits successfully for a oneshot service, so the success ping fires only when the job worked. ExecStopPost= runs after any outcome, and $SERVICE_RESULT lets it report a failure right away instead of waiting for the deadline. Keep the token out of unit files that are world-readable if that matters to you, for example by loading it through EnvironmentFile=.

Create the heartbeat monitor with an expected interval matching the timer (daily, for the example above) and a grace period longer than the job's run time plus RandomizedDelaySec. Heartbeat grace period and interval explains how to choose them, and how to set up a heartbeat monitor walks through creating one.

Option 3: watch the timer from the outside with a server agent

If you already run a server monitoring agent, a down host is flagged on its own, which covers the machine-is-off case. See server monitoring. It does not know whether a particular job ran, so keep the heartbeat for the job itself.

Test the failure path

Make the job fail on purpose once: run sudo systemctl start backup.service with the script temporarily exiting 1, and confirm you get an alert. Then disable the timer with sudo systemctl disable --now backup.timer and confirm the heartbeat alerts after the grace period. Re-enable it afterwards.

Checklist

  • The .timer is enabled and shows a NEXT time in systemctl list-timers.
  • Persistent=true is set for jobs that must not be skipped.
  • The service has a timeout and uses absolute paths and an explicit User=.
  • A heartbeat ping is sent on success and a /fail ping on failure.
  • The heartbeat grace period covers the longest normal run.

Validate the schedule

If you are converting an old crontab line to OnCalendar, check the schedule you meant to write first.

Cron expression explainerPaste a cron expression and see in plain language when it runs, before translating it to a timer.

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading