Why synthetic alerts fire at random

Our synthetic alert libraries fire at semi-random times instead of on a fixed, cron-like schedule. It’s a decision that might seem a little strange at first, so this page explains why randomness is important in such a system, why the kind of randomness we chose is the right kind, and why it needs bounds to avoid false alarms. The reasoning borrows from decades of internet measurement practice. For the schedule itself and its defaults, see Synthetic alert libraries.

Why sample at random

A synthetic alert is a fire drill for your alerting pipeline, and each firing is a sample: it checks that the pipeline works at one particular moment. The question is which moments to check.

The obvious answer is a clock: fire at the top of every hour, hold the alert for ten minutes, then let it resolve. The trouble is that pipelines have rhythms of their own. Batch jobs run nightly, logs rotate, deploys go out in a window, and scrapes and rule evaluations tick on fixed intervals. A drill on a fixed schedule lands at the same point in each of those rhythms every time. If something breaks alert delivery only at 2:30 a.m., a drill at the top of every hour passes forever.

Network measurement hit this problem first. RFC 2330, the IETF’s framework for measuring internet performance, warns that when the thing being measured has periodic behavior of its own, periodic sampling may “observe only part of the periodic behavior if the periods happen to agree.”

Why Poisson

Random gaps solve the rhythm problem, but the kind of randomness still matters. Poisson sampling separates samples with exponentially distributed gaps. Those gaps are memoryless: however long it has been since the last firing, the wait for the next one looks the same, so the schedule hides no period of any length. RFC 2330 says Poisson sampling “can be used to accurately collect measurements of periodic behavior,” and concludes that “we in general prefer that samples of Internet measurements are gathered using Poisson sampling.”

A drill has a second reason to want exponential gaps. Much of what breaks an alerting pipeline breaks after a quiet stretch: an idle connection times out, a credential expires, a pooled mail session goes stale. Exponential gaps vary widely, from a few minutes to a few hours, so over time the drill tests the pipeline after short and long quiet stretches alike. Gaps bunched tightly around one length would test the same quiet stretch over and over.

Poisson sampling also comes with a guarantee that queueing theory has a name for, PASTA: Poisson arrivals see time averages. Poisson firings find the pipeline in each of its states about as often as the pipeline spends time in that state, so over many drills they see the pipeline as it typically is, not as it is at some special moment.

Why alerting needs bounds

Pure Poisson sampling has a catch for alerting. Exponential gaps have no upper limit. Most are short, but now and then one runs very long, and RFC 2330 itself notes that the exponential distribution “is unbounded, so its use will on occasion generate lengthy spaces between sampling times.”

Long gaps are a problem for a check-in timer. You set the timer’s maximum age, how long the phone app waits for a check-in before raising an alarm, and a gap longer than the maximum age raises an alarm for a pipeline that is fine. With unbounded gaps, that false alarm is a matter of when, not whether.

False alarms are not harmless. Each one wakes someone up, or pulls them away from their evening, over a pipeline that was fine, and that wears on the people who receive them. Each one also teaches them that the alarm can be ignored, and an alarm people ignore protects nobody. Triple Pat exists to help techies rest easy, and a drill that raises false alarms works against that directly.

The measurement community ran into the same thing. RFC 7312, which updates RFC 2330’s sampling framework, has a section titled “When to Truncate the Poisson Sampling Distribution.” It describes a path element “with reactive behavior sensitive to flow inactivity” that “could change state if the random inter-packet time is too long,” and recommends that you “truncate the tail of Poisson distribution when needed to avoid reactive element state changes.” A check-in timer is exactly that kind of element: it changes state after too long without a check-in. So the libraries draw every gap from the part of the exponential distribution below a maximum, 2 hours by default. You then set the timer’s maximum age longer than one firing plus the longest gap plus your pipeline’s latency. With the defaults, 3 hours is comfortable.

The libraries also set a minimum gap, 10 minutes by default. That bound is not about false alarms. It makes sure the alert visibly resolves between firings, so every firing is a fresh transition from quiet to firing, the same transition a real incident takes.

Why mostly Poisson is good enough

Bounding the gaps has a cost. Right after a firing ends, the minimum gap rules out another one, and near the 2-hour maximum, the next firing is no surprise. The gaps are only roughly memoryless, and the firings only roughly a Poisson process.

The guarantee survives anyway. In “The Role of PASTA in Network Measurement” (SIGCOMM 2006), Baccelli, Machiraju, Veitch, and Bolot show that seeing time averages does not require exact Poisson. Any mixing process gets the same guarantee, as long as the sampling does not disturb what it measures. Mixing means, roughly, random enough that the distant past says nothing about what comes next. They call the result NIMASTA, non-intrusive mixing arrivals see time averages, and their talk lists truncated Poisson by name as an example. A drill adds about one alert an hour to a pipeline that handles many more, so it disturbs very little.

The same authors argue that Poisson is not required, and for estimating an average with low variance they recommend gaps bunched tightly around one length. That is good advice for measurement. A drill is not estimating an average, though. It is looking for the quiet stretch that breaks delivery, and tightly bunched gaps would never produce one. NIMASTA shows that other distributions can work in theory. Truncated Poisson has the long history: the IETF has recommended Poisson sampling since 1998, and RFC 7312 notes that in measurement practice, truncating it “has been used without issue.”

You can move closer to the ideal by lowering the minimum gap and raising the maximum, but each has a cost. A lower minimum gives the alert less time to resolve between firings. A higher maximum needs a longer maximum age, and a longer maximum age means a real outage in your alerting goes unnoticed for longer. The defaults are in the reference.