Check in from the Watchdog alert

How do you know your alerting works? A quiet pager can mean everything is fine, or it can mean your alerting is broken. If you run Prometheus, your cluster probably already ships half of the answer. The Watchdog alert fires forever on purpose, waiting to be pointed at something that notices silence. A Triple Pat timer is that something.

A check-in timer is what embedded systems people call a watchdog timer, and a check-in is the kick. Point the Watchdog alert at a timer and every delivered notification becomes a check-in. If the notifications stop, the timer raises an alarm, which is the one thing a broken alerting pipeline cannot do for itself. You may have heard this pattern loosely called a dead man’s switch, though the mechanics here are those of a watchdog timer or simply checking in regularly.

This guide is written for the person who runs the monitoring but cannot ship changes to the applications it watches. You want the guarantee, and you should not need anyone else’s code review to get it. Everything below is Alertmanager configuration you already control. No application changes, no new metrics. You will need a timer UUID of your own, and installing the phone app gets you one. If your cluster runs kube-prometheus-stack, expect this to take about ten minutes.

The alert you already have

kube-prometheus-stack ships this rule, enabled by default:

- alert: Watchdog
  expr: vector(1)
  labels:
    severity: none
  annotations:
    summary: An alert that should always be firing to certify that Alertmanager is working properly.

vector(1) is a constant, so the alert is born firing and never resolves. Its own description says what it is for: “This is an alert meant to ensure that the entire alerting pipeline is functional. This alert is always firing, therefore it should always be firing in Alertmanager and always fire against a receiver.”

No kube-prometheus-stack? The YAML above works verbatim inside any Prometheus rule group. Add it and you have a Watchdog.

Route the Watchdog to a timer

Here is the invariant you are about to build. As long as Alertmanager can deliver the Watchdog, your timer sees a check-in every six minutes or so. If 15 minutes ever pass without one, something in the delivery path has stopped, and the timer alarms.

If you run kube-prometheus-stack, open your values file and find the alertmanager.config key. The chart’s default configuration already routes the Watchdog — to a receiver named null that discards it:

route:
  routes:
    - receiver: 'null'
      matchers:
        - alertname = "Watchdog"

The route is already waiting. Point it at a timer instead:

route:
  routes:
    - receiver: triplepat
      matchers:
        - alertname = Watchdog
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 5m

receivers:
  - name: triplepat
    webhook_configs:
      - url: https://triplepat.com/api/v1/checkin/YOUR-TIMER-UUID

Three settings do the work. group_wait: 0s sends the first notification immediately. repeat_interval: 5m re-sends it for as long as the alert fires, which for the Watchdog is forever. One subtlety is worth knowing so the numbers below do not surprise you: Alertmanager re-delivers on the first group_interval tick after the full repeat interval has elapsed, so the cadence is the repeat interval plus one tick — six minutes with these settings, and exactly the spacing we observed. That is your check-in cadence.

The webhook body is ignored: any request to the check-in URL is a check-in, so there is no payload template to write — the URL is the whole integration (details in the check-in API reference). The UUID works like an API key, and it is fine in a receiver URL; Alertmanager configuration is admin-visible anyway.

Now watch it work. We ran exactly this configuration against the deliberately public timer from the tutorials:

curl https://triplepat.com/api/v1/getlastcheckin/f81d4fae-7dec-11d0-a765-00a0c91e6bf6

{
  "lastCheckinTime": "2026-08-23T13:46:22Z",
  "uuid": "f81d4fae-7dec-11d0-a765-00a0c91e6bf6"
}

One cycle later, the timestamp had moved:

{
  "lastCheckinTime": "2026-08-23T13:52:20Z",
  "uuid": "f81d4fae-7dec-11d0-a765-00a0c91e6bf6"
}

Every delivered notification is now a check-in.

Set the timer’s alert interval

In the app, set the timer to alert when 15 minutes pass without a check-in — three times the repeat interval. With deliveries about six minutes apart, one transiently missed delivery leaves a twelve-minute gap and pages no one; two missed in a row means the pipeline is genuinely quiet, and the timer alarms.

Prove it

Break delivery on purpose. Silence the Watchdog in the Alertmanager UI, or with amtool:

amtool silence add alertname=Watchdog --alertmanager.url=http://localhost:9093 --duration=30m --comment="fire drill"

Point --alertmanager.url at your Alertmanager.

The alert still fires, but nothing is delivered, so the check-ins stop. Thirty minutes of silence leaves a gap far past the 15 minute threshold, so the alarm must fire on your phone. Do not be tempted to shorten the drill: a silence much shorter than the threshold can end before the gap crosses it. Our first attempt used an eight minute silence and produced a fourteen minute gap, one minute shy of an alarm. When the silence expires, delivery resumes on the next group-interval tick — in our run the next check-in arrived within a minute — and the alarm clears.

What this proves, and what it does not

Every six minutes, this drill proves a specific chain: Prometheus evaluated a rule, Alertmanager accepted the firing alert, your route selected the receiver, and the notification was delivered. That is more than most alerting setups can say about themselves.

Two things it cannot prove. vector(1) needs no scraped data, so a dead collection pipeline leaves the Watchdog firing happily. And the Watchdog is born firing, so it never exercises the quiet-to-firing transition a real incident takes. Drilling those requires emitting a synthetic alert metric from the systems being monitored, then alerting on that metric instead of the Watchdog. If your services are in Go, our go-syntheticalert-prometheus library emits a suitable metric in one goroutine; guides for emitting and alerting on it are being written.

Next steps