What makes Triple Pat reliable?

Your alerting system cannot tell you that your alerting system is broken. That is the gap Triple Pat fills, and we can only do that if we stay up when other things do not. For Triple Pat, reliable means that when a client tries to check in, there is somewhere for that check-in to go. So why do we think there always will be?

The short answer is that a check-in has ten HTTPS front doors on five servers, plus an email path that reaches the same five, and any one of them suffices. An outage big enough to close them all is an outage you would already know about. We have run for more than two years without a service-wide outage, which is encouraging, but a track record is evidence and not an argument. The rest of this page is the argument.

What has to fail

Every check-in server accepts check-ins and reports the same state, so a check-in succeeds if it reaches any one of them. The design page lists the five servers and where they live. The summary is three cloud providers, five regions in three countries on two continents, IPv4 and IPv6 on every host, and two domains served by two DNS providers. A client can rotate through the ten HTTPS endpoints, or send one email to UUID@checkin.triplepat.com. Both domains list all five hosts as mail servers at equal priority, so the sending mail server tries them in random order until one accepts.

For a check-in to have nowhere to go, one of the following has to be true:

All five servers are down at the same time, in five separate failure domains at three competing providers in three different countries.
Your network cannot reach any of the five locations over either IPv4 or IPv6.
GoDaddy and Cloudflare are both unable to serve DNS at the same time.

None of these is impossible. The first and third are the kind of event that makes the news. The second is more local, and it is a failure that deserves an alarm, because a network that cannot reach five separate locations on two continents over either protocol is unlikely to be reaching your pager provider either. Which brings us to the point of the exercise: if things are broken badly enough to close every one of our front doors, they are broken badly enough that your real alerts are firing, your dashboards are red, and your phone is already ringing. You do not need a fire drill to prove your alarm works while the building is actually on fire.

Why the copies fail independently

Five servers are only worth more than one if they do not fail together, so we bought them from companies that are fighting each other for the same customers. Hetzner, AWS, and Google Cloud have every incentive to keep their infrastructure separate from one another, and GoDaddy and Cloudflare likewise. Competition does not guarantee that they never fail together, but it does give us separate failure domains without our having to build or audit the separation ourselves. A failure inside one provider is unlikely to reach the other two.

The same logic gave us two domains, held at two registrars and served by two DNS providers, so a dispute with one company or an outage at one provider cannot take both. We also registered them at different times of year, so their expirations never line up.

An honest reliability page has to say what the copies have in common, because those are the failures that would take down more than one server at once.

Certificates. Every host gets its certificate from Let’s Encrypt. A Let’s Encrypt outage does not stop a host that already has a valid certificate, but a fleet-wide renewal failure would eventually close every HTTPS front door. We alert on any certificate that is within three weeks of expiry, which is long enough to fix a broken renewal by hand.
Synchronization. The servers exchange check-ins over WireGuard tunnels that Tailscale sets up between every pair of hosts. Tailscale’s own service only does the introductions. Once a tunnel is up it runs directly between the two machines, so a Tailscale outage mostly leaves the existing tunnels working and stops us from adding a new host until it is over. A single tunnel that did go down would not stop anything, because every server has multiple neighbors, so the updates would come in via the other servers.
Software. All five servers run the same code, so a bug ships to all five. The deployment section below is about making that as hard as we can.
The deploy pipeline. Deploys run through GitHub Actions. If GitHub is down we cannot deploy, which is a nuisance for us and no risk to you.

The full list of what we depend on and why is in our post on dependencies.

Why every server can be a full copy

Most databases spend enormous effort making sure two servers never disagree about a write. We get to skip all of it, because of what we store. The state of the service is one fact per timer, the time of its most recent check-in as stamped by the server that received it, so client clocks never enter into it. Merging two copies of that state means taking the later time for each timer, and nothing else. That merge gives the same answer no matter which order the copies are combined in, no matter how many times, and no matter how long they were apart.

In the literature this is a conflict-free replicated data type, and the properties of the merge itself can be established mathematically rather than by testing. That does not make the whole service proven correct, since clocks, disks, and networks still have to do their jobs, but it does remove one of the hardest problems in running a distributed database. For us it means no leader election, no quorum, and no server that is special. Every server accepts writes, every server serves reads, and the service works as long as one of them does. The design page has more on this choice.

Concretely, suppose two of the five servers lose contact with the other three. All five keep accepting check-ins, and each side ends up with a complete copy that is missing whatever the other side received. When the link comes back, the servers exchange their full state and each takes the later time for every timer. No server has to decide which side was the real one, and no check-in ever waited for a quorum. The phone app fails in the same direction. If it reads from a server that has not yet heard about a check-in, or cannot reach any server at all, it keeps the last state it saw and raises an alarm when that state expires. A split can produce an alarm you did not need. It cannot produce silence you did not want.

Operators are the biggest risk

Once a system is this redundant, the most likely way to take it down is for one of us to do it by accidental misconfiguration. So our deployment process is built on the assumption that the next release might be the bad one, and it is deliberately slow. We never update more than one machine at a time, and we do not move to the next machine until the one we just updated reports that every service is running and healthy. A bad configuration or a bad build can take down one server. It should not be able to take down two, and it has never taken down five.

Our record so far

The arguments above are the reason to expect reliability. Our status page, run by a third party that probes every host from outside, is the evidence. In the two years the service has been running we have upgraded operating systems, moved between cloud providers, destroyed and rebuilt servers from scratch, added IPv6 to the whole fleet, and deployed new releases more than a hundred times. Individual hosts have had bad days, and each one has an incident report. Through all of it, every probe of the entire fleet has found at least one front door open.