Heartbeat monitors: how to choose the interval and grace period

Set the expected interval and grace period of a cron job heartbeat so a late run doesn't page you at 3am but a dead job is still caught the same morning.

· 6 min read · By the Spot Downtime team

A heartbeat monitor has two numbers to set: how often the job should check in, and how long to wait after that before raising an alert. Get them wrong in one direction and you are paged for jobs that were merely slow. Get them wrong in the other and a dead backup goes unnoticed for a day. This post explains how they work together and how to pick values for real jobs.

If you haven't created a heartbeat yet, start with how to set up a heartbeat monitor.

What the two settings mean

  • Expected interval: how often your job runs. In Spot Downtime this ranges from every minute to once a week.
  • Grace period: extra time allowed after the expected moment before the monitor counts as down. It can be none or up to 24 hours.

The monitor is down when no ping has arrived within the interval plus the grace period, or when your job calls the /fail URL. So a nightly job with a 30 minute grace period is only reported after the next ping is 30 minutes overdue.

Choosing the interval

Set the interval to the schedule of the job, not to how fast you would like to hear about problems. A job that runs once a day pings once a day. Setting a one hour interval on it would alert you 23 times for a healthy job.

Match the cron expression exactly. If you aren't sure what an expression means, the cron expression tool shows the next run times.

bash
# Every 15 minutes   -> interval: 15 minutes
*/15 * * * * /opt/sync.sh && curl -fsS -m 10 --retry 3 https://example.com/ping/TOKEN

# Daily at 02:00     -> interval: 1 day
0 2 * * * /opt/backup.sh && curl -fsS -m 10 --retry 3 https://example.com/ping/TOKEN

Irregular schedules (weekdays only, first of the month) don't fit a fixed interval cleanly. Use the longest gap in the schedule, such as a Friday to Monday gap of three days, or split the job into separate heartbeats.

Choosing the grace period

Grace covers the variation in when the ping arrives. There are three sources of variation, and each one tells you how much to add:

1. Run time of the job

If the job pings at the end, the ping time is start time plus run time. A backup that usually takes 8 minutes but takes 25 on the night of a month-end close needs a grace of at least 25 minutes plus some margin. Look at your longest recent runs, not the average.

2. Scheduler jitter and load

Cron starts jobs within a minute of the schedule, but a queue worker or a CI scheduler under load can start minutes late. A small grace of a few minutes absorbs this without hiding real failures.

3. Retries

If the job retries on failure, the ping arrives later in the retry case. Grace should cover one retry cycle if you want to be told only when the retries also failed.

A workable starting rule is the longest normal run time, plus 50 percent, rounded up. Then tighten it after a few weeks of real data.

Examples

  • Every minute queue consumer. Interval 1 minute, grace 2 to 5 minutes. Short grace is fine because a stalled consumer is urgent.
  • Hourly report that takes up to 10 minutes. Interval 1 hour, grace 15 minutes.
  • Nightly database backup that takes 20 to 40 minutes. Interval 1 day, grace 1 hour. You hear about a missed backup within the first hour of the morning, long before anyone needs it.
  • Weekly invoice run. Interval 7 days, grace a few hours. Missing it is costly, so don't make the grace generous just to avoid noise.

Ping at the end, and also signal failure

Pinging only on success means a crash and a hang both end as silence, which the grace period then turns into an alert. Calling /fail from the error path marks the job down immediately instead of waiting out the interval plus grace.
bash
#!/bin/bash
PING=https://example.com/ping/TOKEN

if /opt/backup.sh; then
  curl -fsS -m 10 --retry 3 "$PING"
else
  curl -fsS -m 10 --retry 3 "$PING/fail"
fi

Common mistakes

  • Grace longer than the interval. A job that runs every 10 minutes with a 2 hour grace takes more than 2 hours to be reported. Keep grace well under the interval for frequent jobs.
  • Timezone drift. A job scheduled in local time shifts by an hour around daylight saving changes. See cron timezone and DST pitfalls.
  • Overlapping runs. If a run starts before the last one finished, pings bunch up and hide a stuck job. See cron job overlapping runs.
  • Pinging at the start. A ping at the start of the script proves cron fired, not that the work finished.
  • One heartbeat for several jobs. One job's success hides another's failure.

Test your numbers

After you set the values, stop the job on purpose once in a non-critical environment, or pause the cron entry, and confirm the alert arrives when you expect. Cron job heartbeats are on the Starter and Pro plans; see the cron job monitoring page for details.

Cron expression toolPaste your schedule and see when it will next run, so the interval you set matches it.

More on silent failures is in why cron jobs fail silently.

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading