How to monitor cron jobs (and why they fail silently)

Most monitoring answers the question “is this server responding?”. Cron jobs need the opposite question: “did something that should have happened, happen?” Nothing connects to a cron job, so there is no endpoint to poll and no error to catch. When a nightly backup stops running, the silence looks exactly like success.

The four ways a cron job fails without telling you

• It never starts. The crontab entry was lost in a server migration, the user account was disabled, or cron itself is not running. There is no log line for a job that was never invoked.
• It starts and dies immediately. Cron runs with a minimal environment — often no PATH beyond /usr/bin:/bin, and none of your shell profile. A script that works when you run it by hand fails with “command not found” under cron.
• It runs but does nothing useful. The job exits 0 after failing to find any work, because an upstream path changed. Technically it succeeded.
• It runs forever. The previous invocation is still going when the next one starts, they contend for the same lock or table, and eventually the box runs out of memory.

Heartbeat monitoring: invert the check

Instead of something reaching in to check the job, the job reaches out to say it finished. You get a unique URL, the job calls it on success, and the monitor alerts when a call does not arrive within the window you expect. The absence of a signal is the signal.

The important detail is placement: put the call at the very end of the job, after the work, so it only fires when the work actually completed.

# Wrong: fires even if the backup failed.
0 3 * * * curl -fsS https://api.hypermonitors.com/v1/heartbeat/YOUR_TOKEN; /usr/local/bin/backup.sh

# Right: only fires when backup.sh exits 0.
0 3 * * * /usr/local/bin/backup.sh && curl -fsS https://api.hypermonitors.com/v1/heartbeat/YOUR_TOKEN

Inside a script rather than a crontab line, the same idea with the exit code preserved:

#!/usr/bin/env bash
set -euo pipefail

HEARTBEAT="https://api.hypermonitors.com/v1/heartbeat/YOUR_TOKEN"

do_the_work

# --max-time so a hung request can never wedge the job itself.
curl -fsS --max-time 10 --retry 3 "$HEARTBEAT" > /dev/null

Choosing a grace period

The grace period is how long after the expected time the monitor waits before calling it down. Too tight and a job that occasionally runs long pages you at 3am for nothing. Too loose and a genuinely dead job stays undetected for hours.

A workable rule: take the job's normal runtime, add the interval, then add half again as headroom. A backup that runs hourly and normally takes four minutes gets a grace period of roughly ninety minutes — long enough to absorb a slow night, short enough that a missed run is caught before the next business day.

What heartbeats do not catch

Be clear about the limits. A heartbeat proves the job reached its final line — not that it did the right thing. A backup that writes a zero-byte file and exits 0 will happily ping you every night. If that distinction matters, assert on the outcome before pinging: check the file size, count the rows, verify the archive opens. Then the heartbeat means something.

pg_dump mydb | gzip > /backups/db.sql.gz

# Only ping if the dump is plausibly sized (here: over 1 MB).
if [ "$(stat -c%s /backups/db.sql.gz)" -gt 1000000 ]; then
  curl -fsS --max-time 10 "$HEARTBEAT" > /dev/null
fi

Containers, Kubernetes and serverless

The pattern is identical wherever the job runs, which is the point — a heartbeat needs only outbound HTTP. A Kubernetes CronJob pings from its last container command; a Lambda pings at the end of the handler; a GitHub Actions workflow pings in a final step with `if: success()`. Nothing needs to reach in through your firewall.