How to set up a heartbeat monitor for cron jobs in Spot Downtime

A step-by-step guide: create a heartbeat, copy its ping URL, call it from your job, report failures right away, and test the whole thing before you rely on it.

· 5 min read · By the Spot Downtime team

A heartbeat monitor watches something that has no URL to check: a cron job, a backup script, a queue worker. Instead of us calling it, it calls us each time it finishes. If the call doesn't arrive in time, you're alerted. (For why this pattern catches jobs that crash, hang or never start, see how to monitor cron jobs.)

This guide walks through setting one up in Spot Downtime, from creating it to testing that the alert really fires.

The deadline is the last ping, plus the interval, plus the grace period.
Heartbeat monitors are part of the Starter and Pro plans.

Step 1: create the heartbeat

  1. Open Monitors and click Add monitor.
  2. Choose Heartbeat.
  3. Enter a Name that says what the job does, such as Nightly database backup. It's the first thing you'll read in the alert.
  4. Set Job runs every to the job's schedule: anything from 1 minute to 1 week.
  5. Set the Grace period: how late the job may check in before you're alerted (more on choosing it below).
  6. Click Add monitor.

Step 2: copy the ping URL

Open the new monitor. The Ping URL card shows its unique address, with a Copy button:

text
https://spotdowntime.com/api/v1/ping/YOUR_TOKEN
The monitor page shows the last ping and the URL to copy into your job.

The token in the URL is the only key, so treat it like a password: keep it in an environment variable or your CI's secrets, not in a public repository.

Step 3: call it from your job

The URL accepts a few kinds of request:

RequestMeans
GET, POST or HEAD …/TOKENThe job ran. Marks the monitor up and starts the countdown to the next deadline.
GET or POST …/TOKEN/failThe job failed. Marks the monitor down and alerts right away, without waiting for the deadline.

Crontab

Ping on success, report failure otherwise:

crontab
0 2 * * * /opt/backup.sh && curl -fsS -m 10 --retry 3 "$HEARTBEAT_URL" || curl -fsS -m 10 "$HEARTBEAT_URL/fail"

-m 10 means a slow network can never hang your job, and --retry 3 rides out a brief blip.

A shell script

bash
#!/usr/bin/env bash
set -euo pipefail
trap 'curl -fsS -m 10 "$HEARTBEAT_URL/fail" || true' ERR

pg_dump "$DATABASE_URL" | gzip > /backups/db-$(date +%F).sql.gz
aws s3 cp /backups/db-$(date +%F).sql.gz s3://my-backups/

curl -fsS -m 10 --retry 3 "$HEARTBEAT_URL" || true

Any failing command triggers the trap, which reports /fail. The || true makes sure monitoring can never break the job itself.

Node.js

js
try {
  await runNightlyExport();
  await fetch(process.env.HEARTBEAT_URL, { signal: AbortSignal.timeout(10_000) });
} catch (err) {
  await fetch(process.env.HEARTBEAT_URL + "/fail", { signal: AbortSignal.timeout(10_000) }).catch(() => {});
  throw err;
}

More examples, including Python, GitHub Actions and Kubernetes CronJobs, are in the heartbeat docs.

Step 4: choose the grace period

The deadline is the last ping plus the interval plus the grace period. Set the grace period to the job's longest normal run time, plus a margin:

JobJob runs everyGrace period
Nightly backup, usually 20 minutes1 day1 hour
Hourly report, a few minutes1 hour15 minutes
Sync every 15 minutes15 minutes5 minutes
Queue worker pinging while healthy1 minute1 to 5 minutes

Too short and a job that runs a bit long sends a false alarm; too long and you hear about real failures late. After a week, look at the monitor's ping history and tighten it.

Step 5: test it

Don't wait for the first real failure to find out whether the setup works. From any terminal:

  1. Run curl -fsS "$HEARTBEAT_URL". On the monitor, Last ping updates and the status turns up.
  2. Run curl -fsS "$HEARTBEAT_URL/fail". The monitor goes down and your alert channels get a down alert.
  3. Ping the plain URL again. It recovers and you get the recovery alert.

If no alert arrived, check that you have a channel under Alerts and use its Send test button.

Tips

  • One heartbeat per job. Sharing one between jobs means the alert can't tell you which one stopped.
  • Ping at the end, not the start. A ping at the start only proves the job began.
  • Pausing for maintenance? Pause the monitor while you move servers, so the missing pings don't alert.

More in this series

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading