Synthetic alert libraries

The synthetic alert libraries emit a synthetic alert metric from your own code. Your monitoring alerts on the metric like any other, the alert’s notification target is a Triple Pat check-in timer, and every delivered alert becomes a check-in that proves the alerting path it took still works. This page defines the metric every library emits, lists the libraries, and then explains how they work, so you can adapt one to a setup its README does not cover. If your monitoring already has an always-firing Watchdog alert, Check in from the Watchdog alert is a quicker start. The synthetic alert metric goes further, because every firing also takes the alert from quiet to firing, the way a real incident does.

The synthetic alert metric

Every library emits the same metric, so this section applies whichever library you use.

• Name: triplepat_synthetic_alert. OpenTelemetry and Micrometer register it as triplepat.synthetic.alert. Their Prometheus exporters render it with underscores; a backend that keeps OpenTelemetry names as they are shows it dotted.
• Type: gauge.
• Value: 1 while the synthetic alert should be firing, 0 otherwise.
• Schedule: the value alternates between firings and gaps. A firing holds the value at 1 for a fixed length. A gap holds it at 0 for a random length between a minimum and a maximum. Why the schedule is random explains the randomness.
• Longest time between firings: one firing plus the longest gap, 2 hours 10 minutes with the defaults. Set the check-in timer’s maximum age longer than that plus your alerting pipeline’s latency. With the defaults, a maximum age of 3 hours is comfortable.

Every library lets you change the schedule. The defaults are:

Each README gives the option names in its language.

One series per check-in timer

Each library keeps its schedule in memory, inside the object you create from it: a SyntheticAlert, or in Node.js the function syntheticAlert() returns. The process that creates that object is the schedule’s owner, and every process that creates one runs a separate, independent schedule. A check-in timer needs exactly one, so exactly one series of triplepat_synthetic_alert may feed a given timer. There are two ways to get there:

• Emit it from one place: one process, a singleton deployment, or a small exporter of its own.
• Select one series in the alert: if many replicas emit the metric, write the alert rule’s selector to match exactly one series, pinning a label that survives deploys. A StatefulSet pod’s name survives, and so does its apps.kubernetes.io/pod-index label once relabeling copies it onto the series.

Two approaches that look reasonable do not work:

• Pinning a Deployment’s pod name. A Deployment’s pods get new names on every rollout. After the next rollout the rule matches no series, so the alert never fires, nothing checks in, the timer passes its maximum age, and the phone app raises an alarm for a pipeline that is fine.
• Aggregating. max or sum across replicas combines their schedules instead of choosing one, which brings back every problem of many series.

Those problems are why the rule exists. Ten series feeding one timer are ten independent schedules: firings arrive ten times as often and overlap, so the alert rarely resolves, and none of it proves anything that one series does not. For a drill per cluster, use one timer per cluster, each fed by one series.

Libraries

Each library implements the metric above in one language. Its README shows the exact wiring for each metrics client listed here, along with the pitfalls specific to that client.

To install:

• Go: go get github.com/triple-pat/go-syntheticalert
• Python: uv add triplepat-syntheticalert or pip install triplepat-syntheticalert. The package imports as syntheticalert.
• Ruby: bundle add triplepat-syntheticalert or gem install triplepat-syntheticalert
• Node.js: npm install @triplepat/syntheticalert
• Java: add com.triplepat:syntheticalert from Maven Central to your Gradle or Maven build, as the README shows for both.

All five libraries are Apache-2.0 licensed and depend on nothing outside their language’s standard library, except that the Java library uses the JSpecify nullness annotations.

How the libraries work

This section is for adapting a library to a setup its README does not cover, and for understanding why the schedule looks the way it does.

Why the schedule is random

Random gaps keep the firings from falling into step with cron jobs, scrape cycles, or anything else periodic, the way a firing at the top of every hour would, so the drill finds your pipeline as it typically is rather than at some special moment. The minimum gap lets the alert visibly resolve between firings, and the maximum gap bounds how long a healthy pipeline goes without checking in. Why synthetic alerts fire at random explains the design in depth.

One function

Each library comes down to one function that answers “should the synthetic alert be firing right now?” with 1 or 0. In Python and Node.js the object you create is that function; in Go, Ruby, and Java it is the object’s value method. The library starts no thread or timer of its own.

Instead of a background loop flipping the value on schedule, the function works out the current state each time it is called. It keeps the time of the next transition, and on each call it steps through every transition due since the last call, drawing new gaps as it goes. The schedule therefore comes out the same whether the function is read every 15 seconds or once an hour.

Go, Python, Ruby, and Java measure time with a monotonic clock, which never jumps. Node.js reads Date.now(), the system’s wall clock, so a step in the system clock lengthens or shortens the firing or gap in progress, and a large backward step can stretch a gap past the 2-hour maximum.

Beyond the clock, the languages differ only in API shape. Options are durations in the language’s own type in Go (time.Duration), Python (datetime.timedelta), and Java (java.time.Duration), and plain numbers in Ruby (seconds) and Node.js (milliseconds). Invalid options fail when the object is created, never when it is read.

Adapting to your system

Adapting a library means arranging for the schedule’s owner to report the function’s result as triplepat_synthetic_alert whenever the metrics are scraped.

If your metrics client can back a gauge with a callback, register a gauge named triplepat_synthetic_alert and hand it the function. That covers most clients, and it is what each README does first.

If the client has no callbacks, set the gauge from the function’s result just before the scrape is answered. The Ruby README does this for prometheus-client in a single-process server, with a small Rack middleware of its own placed ahead of the exporter.

In a multi-process server, the process that answers a scrape may not be the schedule’s owner, so a callback in the owner may never be called. Under gunicorn any worker may answer a scrape, and prometheus_client answers from per-process files without calling any callbacks. Only the master process exists exactly once, so the master owns the schedule and, since no scrape reaches it, sets the gauge from the function on a short loop. That loop is the one thread this setup needs, and it belongs to your deployment, not the library. The gauge must use multiprocess_mode="livemostrecent", so every scrape reports the master’s latest value as a single series; the default mode exports one series per process, which breaks the one-series rule. The gunicorn recipe has the code.

However you wire it, the one-series rule still applies.