Heartbeat Monitoring for Cron Jobs: Alert Before Backups Fail

Heartbeat Monitoring for Cron Jobs: Alert Before Backups Fail

Heartbeat Monitoring for Cron Jobs: Alert Before Backups Fail

Cron jobs do their work in the background until they don’t—and by the time someone notices, the backup or ETL window is already blown.

Heartbeat monitors are the dead-simple insurance that every scheduled task should have. Here’s how we bake them into PingHarbor and the patterns you can steal.

Where heartbeats usually break

  • Renewed servers or containers where the cron never got re-enabled.

  • Certificates expiring or credentials rotated, causing silent exit codes.

  • Queues clogged so the job hangs but never technically “fails”.

Coverage map

Add heartbeats to anything that should run on a cadence longer than 5 minutes and where failure is not immediately visible.

  • Nightly backups + snapshots.

  • ETL/warehouse syncs.

  • Notification digests + billing exports.

  • Infrastructure housekeeping (log rotation, cache warmers).

Designing the heartbeat

  1. Job sends HTTP POST to PingHarbor heartbeat endpoint at start + success.

  2. PingHarbor expects check-in within a defined window (e.g., every 60 min).

  3. Missed heartbeat triggers alerts with escalating channels (Slack → PagerDuty → ticket).

  4. Optional: attach metadata (duration, records processed) for richer incident context.

Alert strategy

  • 1 miss: Slack DM to owner + ping in #ops-heartbeats.

  • 2 misses: PagerDuty low-urgency incident.

  • 3 misses: auto-create Jira ticket and assign on-call.

Automation examples

  • If a heartbeat misses twice, PingHarbor triggers an MCP tool that checks the job’s container logs and posts the last 20 lines to Slack.

  • For backups, we run a quick S3 object count to confirm whether new files landed; include the result in the alert.

  • Scheduled n8n workflows get a heartbeat too—misses create a GitHub issue so automation owners can jump in.

Checklist

  • Inventory every cron/systemd/n8n job → log owner, schedule, expected duration.

  • Add heartbeat endpoints + secrets as Terraform modules, not ad-hoc scripts.

  • Test monthly by intentionally failing jobs in staging and verifying alerts still route correctly.

  • Document remediation runbooks so whoever catches the alert knows the first three steps.

Heartbeat monitors aren’t glamorous, but they turn “we forgot to rotate backups for three weeks” into a two-minute Slack notification. If you want our Terraform module or alert templates, grab them here or ping me in DMs.