Triple Pat Blog

Memoryless Scheduling Should Be Your Default

peter@triplepat.com (Peter Boothe) — Sat, 04 Apr 2026 00:00:00 +0000

When someone says “run this hourly,” what do they actually mean?

Most developers reach for 0 * * * * in cron, or time.Tick(time.Hour) in Go, or setInterval(fn, 3600000) in JavaScript. These all fire at fixed intervals. But fixed intervals are rarely what you actually want. They’re just what the tools make easy.

The better default is memoryless scheduling: intervals drawn from an exponential distribution. Your task still runs about once per hour on average, but the exact timing is random. This post argues that memoryless should be your default, and fixed intervals should require justification.

What’s wrong with fixed intervals?

Three things:

1. Accidental synchronization. Two systems that both run “every hour” will eventually sync up and hit shared resources simultaneously. This causes thundering herds, lock contention, and correlated failures. The more systems you have, the worse this gets. People try to work around this with hacks like “run this every minute at the 15 second mark” and “run that at the 24 second mark.” But all it takes is one “run this every 17 seconds” and your perfect arrangement is lost.

2. Correlation with periodic behavior. Many systems have periodic patterns—daily traffic spikes, hourly batch jobs, garbage collection cycles. Fixed-interval sampling can systematically miss or over-represent states that correlate with your interval. Your “hourly” health check might always run during a quiet period and miss every traffic spike.

3. Biased sampling. There’s a theorem in queueing theory called PASTA: Poisson Arrivals See Time Averages. It says that if you sample a system at random (Poisson-distributed) times, your samples are an unbiased representation of the system’s time-averaged state. Fixed-interval sampling doesn’t have this property.

What is memoryless scheduling?

What we want is a way to guarantee that events happen at an average of once per hour, but where you cannot predict exactly when the next event will happen even if you know when all the previous events happened. This is called the “memoryless” property, and there’s only one statistical process that has it: the Poisson process.

A Poisson process generates events where the inter-arrival times follow an exponential distribution. In Python, it’s one line:

import random
import time

# Sleep for a random duration averaging 1 hour
time.sleep(random.expovariate(1.0 / 3600.0))

The expovariate function returns exponentially-distributed random numbers. The parameter is the rate (events per second), so 1.0 / 3600.0 gives you an average of one event per hour.

In Go, the equivalent is:

import (
 "math/rand"
 "time"
)

// Sleep for a random duration averaging 1 hour
duration := time.Duration(rand.ExpFloat64() * float64(time.Hour))
time.Sleep(duration)

That’s it. That’s the core of memoryless scheduling.

A production-ready implementation

The one-liner works, but production code needs bounds. True exponential distributions are unbounded—you might get an interval of 10 hours or 10 milliseconds. Operationally, you often want guarantees.

Here’s the implementation we use at Triple Pat. It uses our Clock interface instead of calling time directly, which makes it testable:

package memoryless

import (
 "context"
 "math/rand"
 "time"

 "github.com/triple-pat/go/clocks"
)

type Ticker struct {
 Clock clocks.Clock
 Expected time.Duration
 Min time.Duration
 Max time.Duration
}

func (t Ticker) randomWaitTime() time.Duration {
 var wt time.Duration = -1

 // Resample until within bounds
 for wt < t.Min || (t.Max != 0 && wt > t.Max) {
 wt = time.Duration(rand.ExpFloat64() * float64(t.Expected))
 }
 return wt
}

func (t Ticker) Tick(ctx context.Context) <-chan time.Time {
 c := make(chan time.Time) // Unbuffered is important

 go func() {
 defer close(c)
 for ctx.Err() == nil {
 duration := t.randomWaitTime()
 select {
 case now := <-t.Clock.After(duration):
 select {
 case c <- now:
 default: // Don't block if receiver is busy
 }
 case <-ctx.Done():
 return
 }
 }
 }()

 return c
}

A few design choices worth noting:

Resampling vs. clamping. When we get a value outside bounds, we resample rather than clamp. Clamping (forcing out-of-bounds values to Min or Max) introduces spikes at the boundaries. Resampling preserves the distribution shape within the allowed range.

Unbuffered channel. If the receiver is busy when a tick arrives, we drop it rather than queueing. This prevents ticks from “bunching up” if the receiver falls behind.

Clock interface. Using a Clock instead of calling time.After and time.Now directly means we can test the ticker’s mechanics deterministically without waiting for real time to pass. The patterns compose: clocks make memoryless testable, and memoryless makes distributed systems reliable.

Min and Max are optional. Set Max to 0 to disable the upper bound. This lets you use the same code for bounded and unbounded cases.

Testing randomness

How do you test that your random numbers are actually random? You use statistics.

The mechanical behavior—bounds enforcement, channel semantics, context cancellation—is tested deterministically using a Blocking clock. But to verify that the distribution is actually exponential, we use a Kolmogorov-Smirnov test. It generates 10,000 samples and compares the empirical distribution against the theoretical one:

func TestWaitTimeDistribution(t *testing.T) {
 if testing.Short() {
 t.Skip("skipping statistical significance test")
 }

 cfg := Ticker{Expected: 10 * time.Millisecond}

 const sampleSize = 10_000
 intervals := make([]float64, sampleSize)
 for i := range intervals {
 intervals[i] = cfg.randomWaitTime().Seconds()
 }

 // ... compute mean, lambda, sort, then for each sample:
 // compare empirical CDF vs theoretical CDF (1 - e^(-λx))
 // track the maximum difference ...

 // Critical value for 99% confidence
 criticalValue := 1.05 * math.Sqrt(
 -math.Log(0.005) / (2 * float64(sampleSize)))

 require.Less(t, maxD, criticalValue,
 "K-S test failed: distribution doesn't match exponential")
}

This test will spuriously fail about 1% of the time—that’s what a 99% confidence interval means. So we skip it in -short mode and accept the inherent flakiness. This is a reasonable tradeoff: strong statistical verification without making CI unreliable.

When do you actually need fixed intervals?

Fixed intervals aren’t always wrong. Here are legitimate reasons to use them:

External coordination. If you need to sync with an external system that expects requests at specific times, you need fixed intervals.

Human expectations. Daily reports that arrive “around 9am, give or take a few hours” will confuse people. Some things genuinely need to happen at predictable times.

Rate limiting. If you’re allowed exactly N requests per hour by an external API, you might need fixed intervals to stay within limits.

Debugging. Fixed intervals are easier to reason about when tracking down timing issues.

But notice these are all about external constraints. For internal operations—garbage collection, cache invalidation, health checks, metric generation, background sync—memoryless is almost always better.

Real-world applications

I’ve used this pattern for years across different projects:

Internet speed tests. When measuring broadband quality, you want samples that represent actual usage patterns. Fixed hourly tests might always miss peak congestion. Memoryless sampling gives unbiased measurements. (See mlab-test-runner for an early implementation.)

Synthetic alert generation. At Triple Pat, we generate fake alerts on a memoryless schedule to test that our alerting pipeline works. Because Prometheus scrapes at fixed intervals, making our signal generation memoryless ensures the scraper sees our test alerts with unbiased probability.

Mirror synchronization. Our distributed check-in service has multiple servers that need to sync with each other. Memoryless sync intervals prevent them from all trying to sync simultaneously.

Database cleanup. Expired records need to be deleted, but it doesn’t matter exactly when. Memoryless scheduling spreads the load and avoids coordinated spikes.

The industry is getting this wrong

Every major observability tool—Prometheus, OpenTelemetry, Grafana Alloy—scrapes metrics at fixed intervals. This means the entire industry is collecting potentially biased samples by default.

If your system has any periodic behavior that correlates with your scrape interval, your metrics may be systematically misleading. A service that’s slow for 10 seconds every minute might look perfectly healthy or constantly broken, depending on when your collector happened to start.

The PASTA theorem tells us this is fixable: use Poisson-distributed scrape intervals. But none of the major collectors support this. It’s a gap in the ecosystem.

Summary

Memoryless scheduling is the right default for periodic tasks. Fixed intervals should require justification.

Use memoryless when:

The task is internal (no external coordination needed)
The timing doesn’t need to be predictable to humans
You want unbiased sampling of system state
You have multiple instances that might accidentally synchronize

Use fixed intervals when:

External systems expect requests at specific times
Humans need predictable schedules
You’re working around rate limits
You’re debugging timing issues

The implementation is simple—a few lines of code. The hard part is remembering to use it instead of reaching for cron or time.Tick out of habit.

Use it!

The full implementation is available as a gist. A similar implementation is also available in the M-Lab Go library. Both are designed to be drop-in replacements for time.Tick().

All code in this blog post is CC0, which means you can freely use it however you want.

Happy (randomly-timed) coding!

Design of the checkin service

peter@triplepat.com (Peter Boothe) — Fri, 20 Jun 2025 00:00:00 +0000

“Making a service that is radically simple and reliable”

The critical insight is that a service collecting (timestamp, uuid) pairs that only cares about the most recent timestamp for a given UUID is actually maintaining a crdt. All that work that databases need to do in order to prevent conflicting updates just goes out the window, and we can create a database that can be run in parallel with multiple masters without having to worry about the CAP theorem.

Everything else is a consequence of that one insight, combined with a desire for simplicity across the board. The rest of this document is about how we built the backend system, and the principles we stuck to in the process.

Architecture

We use a microservice architecture. Each service is a separate binary run in its own separate Docker container. There are three main services: the user service, the email service, and the checkin service. We implement and run these services to support the phone apps (which run on iOS and Android on user’s phones) and user checkins (which run inside users’ infrastructure).

The user service is necessarily more centralized because it needs to perform actions exactly once. This means that the user service is inherently less reliable, so we should design the phone apps to not need to contact it more than once a week — they should try every day (or maybe even more often?), but things whould only start going sideways if they have been out of contact for a week.

The checkin service is distributed. Our fleet of 5+ checkin servers operate in master-master mode. Each can be authoritative. The checkin-service must be made reliable. We achieve that reliability through redundancy and running things in multiple clouds. Users should always be able to contact a checkin server to either perform a checkin or to retrieve the time of the last checkin.

The email service is distributed the same way email is distributed: with MX DNS records. Each box in our fleet is running an email server, and the SMTP protocol supports failover from one MX recipent to the next. Receiving check-in emails causes the email service to invoke the check-in service for the appropriate UUID.

Users use {protocol}://{server}/api/v1/checkin/{uuid} to check-in and {protocol}://{server}/api/v1/getlastcheckin/{uuid} to get the last check-in for a given UUID. {protocol} can be http or https. {server} can be any of:

triplepat.com triplepat.net
a.triplepat.com a.triplepat.net
b.triplepat.com b.triplepat.net
c.triplepat.com c.triplepat.net
d.triplepat.com d.triplepat.net

The email service is also distributed, because the way SMTP and DNS MX records work means that we can specify a number of email servers and the senders’ MTAs will automatically fail-over. We run one email-service container on every mirror. This allows users to check-in by emailing {uuid}@checkin.triplepat.com or {uuid}@checkin.triplepat.net.

All our APIs, UUIDs, URLs, and email addresses are case-insensitive.

As much as possible, we store all “brains” and user preferences in the phone apps. That way, the user doesn’t have to contact us very often except for checkins. The checkin server has no knowledge of users or userids. It just stores (UUID, timestamp) pairs. We never trust user clocks, so when users perform a checkin, we store a timestamp generated by the server rather than one from the user’s phone. Our servers’ clocks are synced by our cloud providers. We trust them more than we trust user clocks.

Defaults

We use default, boring technology whenever possible. Every time we are clever, we should know the reason why.

This means we use:

SQLite for the checkin-service database.
PostgreSQL for the user-service database.
Tailscale for internal networking.
Letsencrypt for SSL/TLS certificates.
cert-manager to manage them on each host.
nginx for our public web server and reverse proxy.
Docker for containerization.
Docker Hub for our container images.
Prometheus for monitoring and metrics.
Grafana for monitoring dashboards.
Github for our source code management.
Github Actions for CI/CD.
Go for our server programming language.
Kotlin and Swift for our mobile programming languages.
AWS and GCP for our cloud providers, and we only use tiny vanilla machines to avoid lock-in. We also use TILAA to avoid lock-in to “just the big providers”.
Godaddy for our DNS provider for triplepat.com
CloudFlare for our DNS for triplepat.net
Hugo for our static website generator.

We also use every available linter and formatter to make sure our code, configs, and output are internally clean and consistent and align with the expectations of the outside world. Every lint rule we use is one less rule we need to remember, and human brain space is precious.

Reliability

We acquire reliability via the “cockroach” strategy of many cheap replications around the world. We run multiple instances and we run instances in multiple clouds. To do this, we need to make sure the computers we rent in any given cloud are not too expensive. We also need to monitor our systems because “if you aren’t monitoring, you don’t have uptime”.

Efficiency

We keep our memory requirements low, because the cost of running a single server directly translates to how many servers we can realistically run, and more servers means more reliability. LOCKSS is the order of the day. Right now our services collectively (nginx + Alloy + checkin-service) run fine on 1GB of RAM, which means we pay under $20/server/month. Going to 2GB doubles the per-server cost and 4GB quadruples it, etc. For services that don’t need much CPU, RAM is the cost driver, with network usage lurking behind it. Our cockroach strategy means we need to respect RAM usage.

Storage

We use PostgreSQL for the centralized user-service data. This allows for better concurrency and scalability compared to file-based databases, which is important for the user service’s role.

For the distributed checkin-service, we use SQLite. It is simple, small, allows us to store data in a single file per instance, and we don’t have to write our own serialization code. The synchronization between checkin-service instances is handled at the application level, not the database level.

We avoid using database-specific features where practical to retain flexibility. We use PostgreSQL for the user service because we need to be able to write ad-hoc queries over our set of users, so we need the storage to be safely readable and writable by other processes.

Logging

We log using slog. Because of its flexibility, we can swap out backends if we need to use someone else’s log processing software (e.g. Logrus and the like), but outside of main(), things are logged to slog or they aren’t logged. You should use the slog global logger, or derive your custom logger from it, as every main function should set it up with good settings.

Monitoring and Metrics

We set things up to be monitored with Prometheus. Its pull-based logging semantics make the most sense for us because it means that Alloy can choose how much/often to scrape and we can tune things to make sure we don’t spend more than we intend by sending too much to Grafana. Also, pull-based semantics force us to generate metrics that are sampleable, like counts, instead of ephemeral metrics like gauge values. It is appropriate to force the developer to choose the right metric type, because they are the one who knows best what is being measured.

Strongly prefer Counters and Histograms for metrics instead of Gauges. Prometheus works by sampling metrics, and sampling a counter or histogram coarsens data but sampling a gauge loses data.

Every metric name should lint correctly with promlint. Every metric name should also end with the unit of measure or _total. This helps us build dashboards and combine values in a sensible manner. Use base SI units when possible (“meters” instead of “kilometers”), and strongly prefer counts to gauges.

Docker

Every binary we run in production is run in a Docker container. This means each of them need a Dockerfile in the same directory as their main() function. Please use checkin-service/Dockerfile as an example to get you started. If you do it right, then you can keep compilation times low while still getting the benefits of in-container unit testing.

Every Dockerfile should do one thing. Containers are not VMs and should not be used like VMs. We enforce this by using distroless images where we can and only falling back to Alpine-linux images when we need ad-hoc debug tooling.

Run one process per container, and try and invoke that process without forcing the shell to parse a string to run it (use the array form of ENTRYPOINT). The process should run as nobody or we should have a good reason why it can’t.

Use ENTRYPOINT instead of CMD in the Dockerfiles, because our Docker images are meant to hold and run a single binary.

Build and deploy in different containers using multi-stage builds. Among other things, this keeps the deployed size of our images quite small (under 20MB). Also, deploying a compiler alongside every service is wasteful of disk space and bandwidth, but it also needlessly adds complicating pieces to our images. We keep things simple by removing needless complications.

Every Dockerfile should have a HEALTHCHECK directive in it. This allows deployment to stall until a container is actually working instead of treating container invocation as a “fire and forget” operation. There is a healthcheck directory containing helpful libraries and a healthcheck/check-health binary to perform http-based health-checks (like the ones provided by the healthcheck library). Also, the person building the service knows best what the healthcheck should be, so it is appropriate to make the Dockerfile author design it, and not the deployer.

Each Dockerfile should, as much as possible, start with exactly the same text as the others. In particular, everything up through the running of all the unit tests should be the same. This allows us to only run the unit tests once when we build all our containers, because docker build can use the cache-hit from testing the first to skip the tests for the second.

Health checks

Every daemon run in a dockerfile should support health checks. The simplest implementation of health checks can be found in our internal healthcheck library. This contains a server to report health to the container, a binary to connect to that server and report the health back to docker via its exit code, and a simple interface for things that want to be checked.

All of our docker images should have health checks, one way or another.

It is easy to add a healthcheck to your service by implementing the healthcheck.CheckableService interface which has one method. You should have a health check for every service and pass all such services to the health check server when you start invoke it in main().

We should open-source this small library. It is so useful and good.

Test coverage

All code should be 100% covered by unit tests. Getting uncovered code back to being covered should be considered a high-priority task. 100% coverage doesn’t guarantee no bugs, but it does guarantee that every line of code can be executed without crashing in at least one context. Also, the difference between 100% and 99% is psychologically much greater than the difference between 99% and 98% or 79% and 78% — once you start allowing coverage to slip, it’s hard to get it back up and easy to let it get worse. Beyond simply executing every line, striving for complete coverage encourages writing code that is inherently more modular and designed to be testable. “Keep it 100” and cover everything to make sure things stay working and the design stays good.

Linting

All code is linted with golangci-lint. All code has to lint clean, and this is enforced by our Github Actions. You can put exceptions in with comments, but the exceptions should be few and well-motivated. If we find that a particular linter run by golangci-lint is requiring lots of exceptions, we should decide to either turn that linter off or comply with it everywhere.

All prometheus metrics and Dockerfiles are linted too!

Because we don’t have a big team and are often working under time-pressure and “just trying to get it to work”, we want to use every linter possible. Everything that can help us offload “smell tests” to automated machines is a good idea.

Error handling and recovery

Feel free to use rtx.Must and rtx.ValueOrDie liberally for errors that are genuinely unrecoverable or indicate a programming bug (e.g., parsing a known-good embedded asset). Crashing immediately in these cases makes the problem obvious and prevents potentially corrupted state. These helpers also simplify testing, as you don’t need to write tests for error handling paths that simply crash.

All errors must be explicitly handled. Common handling strategies include:

Crashing: Using rtx.Must or rtx.ValueOrDie for unrecoverable errors, as described above.
Returning the error: Passing the error up the call stack for a higher-level function to handle. This is the standard Go approach and is perfectly acceptable.
Handling the error: Implementing specific logic to recover from the error or take alternative action within the current function.
Logging the error: If an error cannot be handled appropriately by the current function (and returning it is not suitable, perhaps because it’s in a background goroutine or at the top level of a request handler), use slog.Error(...) to record it. This increments a Prometheus counter, making unhandled operational errors visible.

Avoid both logging at error and then returning the same error, as this often leads to duplicate log entries. Choose one appropriate handling strategy for each error and stick to it. If you can handle the error, then it likely should cause a Warn or Info log message because it’s not a true error, it’s just something unexpected. Casually logging at the Error level causes alert spam and operator fatigue.

Command line arguments

We build our Docker images with ENTRYPOINT set, which means we pass in command-line arguments in list form as part of the command: directive in that appropriate YAML file.

Operations

We use GitOps as our operations and deployment strategy. When you merge a new tag to the go repo, an image with that tag is built and pushed to the Docker Hub. We use Github Actions to build and push the images.

Code repos run all tests and linters on every push, and build and push new containers on every tag. Config repos push the configs to production on every push or merge to main. We acknowledge that code is configs and configs are code, so the line is fuzzy, but the categorization in practice should be clear in each case.

Push to prod on merge to main:

server-configs
website

Build+Lint on merge, push on tag:

go
ios_app
android_app

We keep our configurations, as much as possible, in the server-configs repo and deployed as the compose.yml file that is run by docker compose. If we move to k8s, then it will be a different file name, but the principle is the same: one file to organize them all.

Deployments of containers always use --wait to ensure that all the containers are healthy before moving on to the next machine. In this way, bad configs that break containers at start only take out one machine.

We rollback just like we roll forward: push and merge to main.

Security

We use letsencrypt for SSL/TLS certificates. We use cert-manager to manage them on each host. Each host manages its own certificates because then we don’t have to worry about running centralized secret provisioning. All our public services are behind nginx, which terminates SSL/TLS and proxies requests to the appropriate internal service.

Our internal-only services are behind Tailscale. We only share internal services between machines over our Tailscale network.

In general, we prefer a “fail2slowdown” instead of an “ostrich” or “fail2ban” strategy. We want to slow down attackers, but we also want to keep our users happy. We want to be able to detect and respond to attacks, but we also want to be able to detect and respond to normal traffic. Public services should be wrapped in a handler to slow down responses to IPs that have sent us erroneous requests. This prevents us from being overwhelmed by bad requests and prevents rumplestiltzkin attacks. That said, people developing against our service will often make lots of bad requests before figuring out how to make the good ones they want to use, so we don’t want to ban them.

Each bad request from an IP causes all responses to that IP to be slower by 1 second for the next two hours, up to a max added delay of 10 seconds.

We also prevent having too many simultaneous connections (good or bad) from a single endpoint both in our nginx config, and in our Go code.

We add a no-index and no-cache headers to all API responses.

Networking

Internal services are not exposed to the public internet, and are only exposed over Tailscale connections. We use Tailscale for all internal networking between hosts. Within a host, we use docker-compose to run services and it sets up a virtual internal network for them.

We use Godaddy for triplepat.com DNS (both registration and serving). We use Cloudflare for triplepat.net DNS (both registration and serving). We have no CDN and do not want one.

We have matching records for DNS on two different TLDs. We have www., @, MX, a., b., c., and d. records for triplepat.net and triplepat.com. Those TLDs are run by two different providers.

Website

We use Hugo to generate our website. Our website is static. Comments are provided by linking to BlueSky posts.

Time

Programming using time is often terrible for testing and a breeding ground for subtle bugs. As much as possible, we use the clocks package to provide a clock that is easy to mock in tests. We use clocks.System for all time-related operations in production, and we use clocks.Static or clocks.Blocking for all time-related operations in tests. We use the .Sleep(), .Now(), and .After() methods associated with those clocks instead of time.Sleep().

Ideally, time.Now() and time.Sleep() (and the helper methods in time which call .now(), like Until() and Since()) should never appear in the codebase outside of the clocks package and _test.go files.

Memoryless

For jobs that need to happen repeatedly, we want to prevent synchronization. The best way of preventing synchronization is to ensure that the events occur in a Poisson process. For that, we created memoryless.Ticker{}, which has one method, Tick(), which produces a <-chan time.Time just like time.Tick().

Production code should use memoryless.Ticker.Tick instead of time.Tick or time.NewTicker or it should have a good explanation as to why.

Subscriptions

Users log in to our services using “Log in with Apple” or “Log in with Google”. This causes us to create an account for them if they don’t already have one. They then get our free tier of service. In the free tier, you get 3 UUIDs, and your UUIDs get scrubbed and deleted every month. If you want UUIDs to never get deleted, you can subscribe to our paid plan.

If you have a subscription, then you get UUIDs that are not scrubbed and are yours forever.

We have a web-hook in user-service to accept notifications from the Apple App Store about new subscriptions. That web-hook updates the state in our DB, and it is the DB state that we will use for our scrubbing utility. The iPhone app also reports its subscription state and supports sending the purchase receipt to us for verification, so we try to make sure we never do harm to an existing customer.

Summary

Keep it simple, respect CS theory, test everything, lint everything, and use automation everywhere. That’s how we built it.

100% test coverage makes your code simpler and better

peter@triplepat.com (Peter Boothe) — Mon, 14 Apr 2025 00:00:00 +0000

Every line of Go code that runs Triple Pat is tested by some test.

% go test ./... -cover=1 -count=1 \
 | awk '{print $5}' \
 | sort \
 | uniq -c
 41 100.0%

That’s 41 packages, and every line is executed at least once in our automated test suite. Why is this good? Why is this not just a silly exercise in pedantic completionism? There’s two reasons: “the Al Capone theory of sexual harassment” and the value of an intellectual forcing function.

Software correctness and “The Al Capone theory of sexual harassment”

Al Capone went to jail not for being a mobster, but for cheating on his taxes. His dishonesty and criminality in one area implied that he was also criminal in others. “The Al Capone theory of sexual harassment” is that sexist harassers in the workplace are likely to also be dishonest and bad for the business in many ways, and should be gotten rid of ASAP. Code that is bad is also likely to be bad in many ways.

The most basic test is the smoke test, where you “turn it on and see if it catches fire”. In a software context, this means running the code and verifying that it didn’t cause a crash. This is a valuable first step towards correctness, especially for greenfield development, because bad code is often bad in many ways (like Al Capone was!). Even the most basic smoke test or in-out test is enough to uncover very real problems.

As code ages and corner-cases are uncovered, the tests will naturally become more complex. But to start out it’s often enough to just make sure nothing crashes by fully covering the code with the most basic of tests.

Tests as intellectual forcing functions

Let’s start with a quote from Tony Hoare:

The real value of tests is not that they detect bugs in the code, but that they detect inadequacies in the methods, concentration, and skills of those who design and produce the code.

When you pursue 100% coverage you design your code for testability. You also find out that many errors are actually impossible (not just unlikely — impossible). A good example is the library google.golang.org/api/idtoken, which has idtoken.NewValidator(context.Context, ...ClientOption) that returns a value and an error. If you don’t pass in any client options, then it is literally impossible for that function to return an error. Therefore, you can (and should, IMO!) use ValueOrDie on its return pair if you aren’t passing in any options.

I would never have known that this was the case if I hadn’t committed to 100% code coverage - I would have just done a mindless error check that returns (nil, err) to the caller. Indeed, that’s the code I wrote before I started trying to test it!

Because the CI coverage report showed this codepath as untested, I tried to cause the error, and I discovered that the error was impossible. This was actually the only error-producing code in the function, so as a result I could also simplify the function I was writing. This insight allowed me to change the code from:

func NewVerifier(ctx context.Context, clientID string) (Verifier, error) {
 validator, err := idtoken.NewValidator(ctx)
 if err != nil {
 return nil, err
 }

 return &verifier{
 validator: validator,
 clientID: clientID,
 }, err
}

to:

func NewVerifier(ctx context.Context, clientID string) Verifier {
 // An error is impossible because
 // we're not using a custom transport.
 validator := rtx.ValueOrDie(idtoken.NewValidator(ctx))

 return &verifier{
 validator: validator,
 clientID: clientID,
 }
}

Pursuit of 100% test coverage forced me to think more deeply about my errors and how to cause them, which led to me discovering that I could make my code simpler, not just in this function but for its callers as well. The simplicity we found “cascaded upwards”! I would never have made this discovery without being forced to, however, because I am a lazy programmer just like everyone else. It was easy to write that error pass-through, so that’s what I did and that’s what most programmers would do. Without this goal of 100% coverage forcing me to think about things, I would not have thought hard enough to notice the opportunity for simplification.

When you think about your code, your code becomes better. The problem is that thinking is hard and tiring, so you need a reason to do it. I recommend making 100% test coverage one of your reasons.

TL;DR.

Every line of code should be executed at least once by your tests. 100% coverage doesn’t mean “no bugs” of course (and for code that needs to accomplish a task, you should definitely also have some assertions about the desired task being accomplished), but full coverage does mean that all remaining crashes are in some sense sneaky. It means they have to be joint test/code bugs, rather than just bugs in the code.

Even “smoke tests” with no explicit assertions implicitly assert that the code they run doesn’t crash. 100% coverage helps keep your code free of simple bugs.

When you try and make sure every line at least gets a smoke test, then you also end up finding that some errors are actually impossible. Impossible errors represent code that you don’t need to write, and complication you can avoid. 100% coverage helps keep your code simple.

100% coverage helps keep code simple and free of simple bugs. Ultimately, the goal is to write correct code, and as Tony Hoare also said:

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

In our twin pursuits of simplicity and correctness, full test coverage plays a valuable role.

Learning in public from our incidents

peter@triplepat.com (Peter Boothe) — Mon, 31 Mar 2025 00:00:00 +0000

Triple Pat runs a service on computers and offers that service to the world. Because computers only mostly work, but we want our service to work all the time, we have to assemble a reliable service out of mostly-reliable components. A critical part of doing that is a post-mortem process where past incidents are analyzed and steps taken to avoid them recurring in the future, so that’s what we do!

Triple Pat runs its post-mortem process largely in the open. We have a monitoring process (run by Better Stack) and they host a public status page for us of the server statuses they measure. The status page can be found at status.triplepat.com. On that main status page, you can see past incidents where some servers weren’t behaving as well as we would have liked.

If you would like to learn more about each incident, you can check out our page of incident reports! Every outage is categorized and described. Some are just acknowledged, but most end up causing changes in how we deploy and run our systems.

From the status page, you can see that not only is our service reliable (because we operate in master-master mode, in order to have an outage, every server must be down simultaneously, and that has never happened), but also that as things happen to our service components, we learn and take action to prevent repeats.

The errors cause both immediate fixes (like rolling back to a previously-working configuration), but they also cause long-term preventions (like adding validation steps to prevent the rollout of broken configs). Reliability comes from good decisions, good decisions come from experience, and experience comes from learning from bad decisions. You can watch along as our organization learns more about where our systems tend to break and what we can do to prevent and/or automatically mitigate that breakage.

We’re proud of our current 100% uptime record for the Check-In service, but we’re even more proud that we’re constantly learning and improving. Doing this helps ensure that our uptime isn’t just luck, it’s the result of systematic learning from past incidents and continuous investment in increased resilience.

Linting your Go code

peter@triplepat.com (Peter Boothe) — Wed, 26 Mar 2025 00:00:00 +0000

You should lint your code! Pretty much everyone agrees that you should do this. Linters are tools of varying degrees of sophistication that attempt to help you avoid dangerous situations with your code, and coding is hard enough that such automated help is invaluable.

The Go programming language was designed to have a relatively simple syntax and to compile quickly, which means it is also straightforward to write a linter for the language. So lots of people did! Eventually their efforts got collected in a meta-linter called golangci-lint. That linter just had version 2 released, so its a good time to talk about its configs.

Here’s the configs for Triple Pat’s Go code. The config is actually a pretty strict one, and I encourage you to be strict when setting up a new project. Linting cleanly is hard to put in an existing project, but is easy to do in the beginning. Start your projects off right!

How does our config differ from yours? What should we do differently?

Reliable DNS

peter@triplepat.com (Peter Boothe) — Tue, 04 Mar 2025 00:00:00 +0000

The Triple Pat check-in service is a service centered around helping people ensure that their software operations alerts are working. This means that we are measuring the reliability of something that is, itself, very reliable. Just like you need a lot of precision to measure something that is very precisely made, you need a lot of reliability to measure a reliable system. Every use of our APIs uses the names of our servers, which means that our system is only reliable as our names are, and our names are provided by DNS.

The Domain Name System (DNS) seems simple in concept: a database mapping names to IP addresses. Unfortunately, there are only two hard things in computer science: cache invalidation and naming things. And DNS is a distributed cached database of names.

This means that, despite superb efforts all around, DNS doesn’t work right sometimes. It’s always in some sense operator error, but those errors happen often enough that we can also blame the tool for providing a gun and aiming it at the operator’s foot. DNS is so often the culprit that we even have websites like isitdns.com and dnshaiku.com. However, at Triple Pat we want to build a reliable system that provides an external API. That API can be accessed in two main ways: by email and by URL. Both methods depend on DNS. So how do we set up our DNS for reliability?

Well, first we have reliability through redundancy. Everything available through triplepat.com is also available through triplepat.net. We have mirrored the records so that if DNS seems down for one, try the other and it should hopefully be up. We enforce the independence of these by buying the names and hosting the DNS through rival companies. GoDaddy is in charge of registration and DNS operations for triplepat.com, and CloudFlare is in charge of registrations and DNS operations for triplepat.net. These two companies are direct competitors, and as such they are unlikely to share infrastructure. So we have achieved independent redundant reliability via incentive-compatibility with large corporations.

Next, for our mail operations to uuid@checkin.triplepat.com and uuid@checkin.triplepat.net, we have MX-type DNS records specifying the mail servers for both checkin.triplepat.net and checkin.triplepat.com. Because we can receive email at any of our mirrors, we make the MX record for each TLD refer to every one of:

and we have made the time-to-live (TTL) for the MX records be as long as each provider allows. This means that email service is highly likely to keep working, even if one TLD goes down, because the long-lived records mean that mail for one can be handled by the other, and that fact can be served out of cache. So, once you have emails flowing, it is highly likely that they will be able to continue to flow even through DNS disruptions.

So we achieve DNS reliability by having multiple providers, and, for the MX (mail) records, by having long-lived cache entries that allow each TLD to back up the other (or not). To see this in action for our systems, use the dig tool!

$ dig checkin.triplepat.net MX
...
;; ANSWER SECTION:
checkin.triplepat.net. 86400 IN MX 10 a.triplepat.net.
checkin.triplepat.net. 86400 IN MX 10 c.triplepat.com.
checkin.triplepat.net. 86400 IN MX 10 b.triplepat.com.
checkin.triplepat.net. 86400 IN MX 10 a.triplepat.com.
checkin.triplepat.net. 86400 IN MX 10 b.triplepat.net.
checkin.triplepat.net. 86400 IN MX 10 d.triplepat.net.
checkin.triplepat.net. 86400 IN MX 10 d.triplepat.com.
checkin.triplepat.net. 86400 IN MX 10 c.triplepat.net.

and

$ dig triplepat.com MX
---
;; ANSWER SECTION:
checkin.triplepat.com. 86400 IN MX 10 a.triplepat.com.
checkin.triplepat.com. 86400 IN MX 10 c.triplepat.com.
checkin.triplepat.com. 86400 IN MX 10 d.triplepat.com.
checkin.triplepat.com. 86400 IN MX 10 a.triplepat.net.
checkin.triplepat.com. 86400 IN MX 10 b.triplepat.com.
checkin.triplepat.com. 86400 IN MX 10 b.triplepat.net.
checkin.triplepat.com. 86400 IN MX 10 d.triplepat.net.
checkin.triplepat.com. 86400 IN MX 10 c.triplepat.net.

From this we can see both sets of names in both responses. Our DNS entries for a,b,c, and d are shorter-lived because we want to be able to migrate servers in less than a day, but the MX record means that the mail sender can failover between TLDs transparently to you, the service user.

No matter which TLD you use for the email, as long as the MX record is in the cache (and to get in the cache, it needs to have worked once in the past 24 hours), your email will get routed to either of the two TLDs.

Even better, it is highly likely that at least one TLD is functioning properly, because the providers are independent. This means that if one of them is functioning correctly, the service should still work. If we find out that two are not enough, then we can easily add a third TLD, but for now we feel pretty confident that two independent DNS registrars/providers is enough to get us the nines we need to provide a reliable service.

Highly reliable Go code - use clocks not time

peter@triplepat.com (Peter Boothe) — Fri, 28 Feb 2025 00:00:00 +0000

Q: How can you tell if someone is a programmer?

A: Easy! Say “time zones” and see if they flinch!

Time and clocks always sound straightforward when you don’t think too hard about them. Unfortunately for us, the job of a programmer is to think hard about the things they are building. And one of the hardest things to get right is testing code that depends on time.

Imagine you have a cache that expires items after an hour, or a retry mechanism that waits 30 seconds between attempts. How do you test these without waiting? Your unit tests would take forever to run!

When working with Go, you often see time.Sleep() and time.Now(). In C the equivalents might be sleep() or usleep() and time(). In C++ you should probably use std::this_thread::sleep_for() and system_clock::now(). In Python it’s datetime.datetime.now() and time.sleep().

In all of these functions, you consult data derived from the OS-managed hardware clock. Because that clock is managed outside of your code, you don’t control it! Worse, you are up a creek when it comes to testing. I would like to propose that, instead of using these global functions, you define an interface called Clock and provide at least two implementations: SystemClock and StaticClock.

In Go this would be:

package clocks

type Clock interface {
 Now() time.Time
 Sleep(time.Duration)
}

type SystemClock struct{}
func (SystemClock) Now() time.Time { return time.Now() }
func (SystemClock) Sleep(d time.Duration) { time.Sleep(d) }

type StaticClock struct {
 Time time.Time
}
func (sc *StaticClock) Now() time.Time { return sc.Time }
func (sc *StaticClock) Sleep(d time.Duration) { sc.Time += d }

With these two implementations, you can embed the clock in your code you want to test. Which means that you can cause your code to use the static clock! All of its sleeps will be instantaneous - you can verify using Now() that time has passed, but you will never have to wait for a Sleep() to complete. They have been transformed into additions!

To use this, you’ll need to change the objects you have that use time to have another class member - the clock. Then at test time you can pass in a StaticClock and at runtime you can pass in the SystemClock. By testing with clock objects, you can make your unit tests both faster and significantly less flaky.

Just for completeness, here’s clocks in Python:

import datetime, time

class SystemClock(object):
 def now(self) -> datetime.datetime:
 return datetime.datetime.now()

 def sleep(self, seconds: float):
 time.sleep(seconds)


class StaticClock(object):
 def __init__(self, currentTime: datetime.datetime):
 self.currentTime = currentTime

 def now(self) -> datetime.datetime:
 return self.currentTime

 def sleep(self, seconds: float):
 self.currentTime += datetime.timedelta(seconds=seconds)

Using these clocks, you can decide whether you want the OS to be in control of your code’s perception of time, or whether you want to be in control. I find that in unit tests I almost always want to be in control, while in production I want to hand control to the OS.

This is called “dependency injection” when generalized, and can be of great help in many contexts. In my opinion, clocks are one of the very best dependencies to inject.

A Practical Example

Let’s look at a real-world example. Suppose you have a cache that needs to expire items after a certain duration:

type Cache struct {
 clock Clock
 items map[string]item
}

type item struct {
 value string
 expiresAt time.Time
}

func NewCache(clock Clock) *Cache {
 return &Cache{
 clock: clock,
 items: make(map[string]item),
 }
}

func (c *Cache) Set(key string, value string, ttl time.Duration) {
 c.items[key] = item{
 value: value,
 expiresAt: c.clock.Now().Add(ttl),
 }
}

func (c *Cache) Get(key string) (string, bool) {
 item, exists := c.items[key]
 if !exists {
 return "", false
 }
 if c.clock.Now().After(item.expiresAt) {
 return "", false
 }
 return item.value, true
}

Now testing this cache becomes trivial:

func TestCacheExpiration(t *testing.T) {
 clock := &StaticClock{Time: time.Now()}
 cache := NewCache(clock)

 cache.Set("key", "value", 5*time.Minute)

 // Advance time by 4 minutes
 clock.Sleep(4 * time.Minute)
 value, exists := cache.Get("key")
 if !exists {
 t.Error("Item should still exist")
 }

 // Advance time by 2 more minutes
 clock.Sleep(2 * time.Minute)
 _, exists = cache.Get("key")
 if exists {
 t.Error("Item should have expired")
 }
}

This test runs instantly, is deterministic, and clearly demonstrates the cache’s behavior. Without StaticClock, this test would take 6 minutes!

In production, you would use:

cache := NewCache(&SystemClock{})

Key Benefits

Testability: Tests run instantly and are deterministic
Control: You decide when and how time advances
Isolation: Your code is isolated from system clock changes
Clarity: Dependencies on time are explicit in your interfaces

Use it!

This pattern is your friend whenever you need to:

Test code with timeouts or delays
Verify behavior at specific times or dates
Make flaky time-dependent tests reliable
Speed up tests that would otherwise need to wait

This is not a new trick, but it’s very useful and it is under-used, and it can make your life much better when you need to test code that relies on time.

All code in this blog post is CC0, which means you can freely use it however you want.

Happy coding!

Reliable Systems via Good Metrics

peter@triplepat.com (Peter Boothe) — Fri, 14 Feb 2025 00:00:00 +0000

When building and running reliable systems, you need data to answer three critical questions:

Is the system working correctly now?
How well has it been working over time?
When did things start to go wrong?

Metrics help us answer these questions. Here we focus on Prometheus metrics and some best practices for them.

Prometheus is a pull-based system for specifying, collecting, and saving metrics. It’s pretty easy to set up and very powerful. There are, however, some best practices around metrics and metric names that are easy to get wrong. Worse, changing metric names is surprisingly difficult, because metrics get used in alerts and dashboards, so a rename can easily break “downstream” systems. It is worth it to try and get your metrics right the first time.

A subtle question that people only start asking themselves after using a metrics system for a bit is “What makes a good metric?” Metrics usually have names and values and types, and there are definitely better and worse ways to name, define, and use metrics in your code. As a quick example, if a metric tracks a duration, its name should end with _seconds, and not _ms or _s or _time. More on this later, but these kinds of rules are important, and are easy to get wrong.

We can describe some best practices around metrics (and we will!) but manual review of these best practices is error-prone. When possible, use a linter to check our metrics. This post describes how to use the promlint tool to check your metrics, and why you might really want to use it. We end with some metrics best-practices, including some that unfortunately promlint can’t check, but are good to keep in mind, and then exhort you to use promlint as part of your unit tests.

`promlint` for linting metrics

Use promlint to check your metrics. It’s great. When it complains, it is right and you should fix your metrics. They built a tool with a very low false-positive rate, so you can trust it.

How to use `promlint`

The go library promlint contains two methods of note: New() and Lint(). Because you want to check all the metrics for your application, you should lint your metrics in a unit test inside the main package so that all the libraries main() uses are included. Here’s how I use it to check my metrics.

Now you can test your own metrics, and if you violate the automatically-checkable best practices you will get an error. So let’s look at some of the best practices, both automatically-checkable and not.

Good metric names

A good metric name leaves no doubt about what the metric measures and where it comes from. Right away from this definition of good we get some rules:

A metric should be clear about where in the code it comes from.
A metric name should specify the type of metric.
A metric should be clear about what it measures.
A metric name should include the unit of measurement.

Pleasantly, the Prometheus project agrees with these rules about naming metrics. This means that promlint can check your metrics for compliance with those rules! As a quick precis, here are the rules:

Counter metric names (and only counter names) should end with _total
All metric names should end with a unit of measurement (just before the _total for counters)
Units of measurement should be an SI unit without a multiplier. Use _seconds for seconds and _bytes for bytes, and avoid _milliseconds, _ms, _us, _ns, _µs, _mB, _megabytes, and _kb

The linked page has more, but these are the important ones. promlint will check your metrics for compliance with these rules as much as possible.

Good metric types

There are four types of metrics: counters, gauges, summaries, and histograms. There’s a few more options, but they are more obscure and are easy to get wrong. You should stick to gauges, counters, and histograms, and really you should try to stick to just counters and histograms.

Counters for metrics that are monotonically increasing.
Histograms for metrics that need to track a distribution of values.
Summaries are like histograms, but they are more complex and have more pitfalls. Prefer the simplicity of histograms.
Gauges for metrics that can go up and down. Use sparingly because they have many subtle pitfalls.

Counters

A critical thing to understand about metrics is that they are collected when the collector wants. Your code has no influence on how often the Prometheus collector comes around to collect metrics. This means you need to design your metrics to be informative even when sampled at very different rates. The most common way to do this is to basically always use counters. Increments of a counter are never missed, the collection rate just determines how coarse the sampling rate is. With counters you will never miss a measurement.

Histograms

Histograms are actually counters under the hood, so everything good about counters is also good about histograms. Histograms never miss a measurement, and the collection rate just determines how deep the metrics user can zoom into the data. If they sample every minute, then they can zoom in and see changes every minute. If they sample every hour, then they can’t zoom in as much. Good histogram names are clear about what they measure, and include the unit of measurement, for example http_request_duration_seconds.

If you’re measuring HTTP request durations with http_request_duration_seconds, the histogram will create multiple counters like:

http_request_duration_seconds_bucket{le="0.1"} (total requests under 0.1s)
http_request_duration_seconds_bucket{le="0.5"} (total requests under 0.5s)
http_request_duration_seconds_sum (total duration of all requests)
http_request_duration_seconds_count (total number of requests)

Now you can see how histograms are a cousin of counters, because they literally are a collection of counters, and everything good about counters is also good about histograms.

Summaries

Summary metrics are a cousin of histograms, but they are more clever internally. They attempt to adaptively track the median value of a metric (or the 95%ile, or some other percentile), but the algorithm for reporting a median is either memory intensive (if they save all measurements and report the exact median at scrape time) or imprecise (if they use a streaming algorithm). The mathematical subtleties are beyond the scope of this post, but the bottom line is that you should strongly prefer histograms. The simplicity of histograms is worth a lot.

Gauges

Gauges are the most problematic type of metric, because there are no rules and therefore no guarantees about gauge values. Gauges have no history, only a current value. A gauge that spikes for 1 second every minute at 15 seconds after the minute will either look like it is constantly zero or constantly spiking when it is collected every minute. Worse, whether it looks like zero or a spike depends on when the collector started! Gauges are best used for information that changes very slowly. I use gauges to track the git commit that the server binary was built from, because that number changes changes very slowly (it only changes when I build and deploy a new binary).

Also, you want to be careful to try and use metrics that aren’t ratios. Export numerators and denominators separately. Don’t export a gauge of http_requests_per_minute, export http_requests_total and let the user do the division. This gives the user more flexibility (turning requests per minute into requests per hour or per second seems easy but is not) and avoids issues when the denominator is zero. In general if the word “per” is in the metric name, you are probably doing something wrong, especially if the denominator is a time unit. Export a counter or histogram for the numerator, and then the user can do the rest.

If you want to keep track of a current allocation, it may be tempting to use a gauge. But this ends up having the sampling problem described earlier. Better is to use two counters, one for allocations and for deallocations. The difference between the two counters is the current allocation. This way you don’t have the sampling problem, and you don’t have to worry about missing massive temporary spikes in allocations. Count packets and bytes sent and received instead of reporting bps. Instead of measuring how much water is in a bucket, measure how much water has been added to the bucket and how much has been taken out.

For example, instead of a gauge like memory_usage_bytes, consider using two counters:

memory_allocations_bytes_total (total memory allocated)
memory_deallocations_bytes_total (total memory freed)

This gives you both the current usage (by subtracting) and the historical allocation/deallocation volumes between samples, which a gauge would miss.

By keeping your metrics simple and sample-rate agnostic, you and your metric’s users can change the granularity or adapt them to new situations. This is a huge win. I’ve lost weeks of my professional life to bad metrics in the past, and I’d like to help you avoid that.

Summary

Metrics are important and good!

Tools:

Use promlint to catch common metric mistakes
Run metric checks as part of your unit tests

Naming:

Your metric name should be clear about what it measures and where it comes from
Put units in your metric names
Use SI units without multipliers
Ensure counter names end in _total

Types and Usage:

Prefer counters and histograms over gauges
Prefer histograms over summaries
Design your metrics so that you never want to have “per” in a metric name
Track changes (like total water added and removed) rather than states (current water level)

Use it!

The promlint tool is great, and you should use it! It will catch some mistakes, and thoughtfully addressing its output will make you think about your metrics in a good way. It is also easy to set up, and you can run it as part of your unit tests.

Along the way, please think deeply about the metric names and types you define and use, and prefer Counters and Histograms. Metrics are a very “sticky” part of a project, so getting them right early is a great way to avoid much bigger headaches later.

Highly reliable Go code - Don't ignore errors when you defer, use this pattern instead

peter@triplepat.com (Peter Boothe) — Tue, 04 Feb 2025 00:00:00 +0000

Continuing our series on highly reliable Go code, here is a nice pattern for handling errors when you defer. Let’s start with a common problem. If you are building a Go service, you might have code that looks like this:

s := http.Server{ /* ...setup... */ }
// This returns an error we're ignoring!
defer s.Shutdown(ctx)
s.ListenAndServe()

Improving `http.Server` usage

There’s a problem here: two functions return error, and we’re ignoring both of them! We can do better.

For ListenAndServe, the fix is straightforward — we should use LogOnError. You can go to the linked post to learn more, but the key is that LogOnError is a function that takes an error and logs it if it’s not nil. The deferred Shutdown is trickier though. Here’s why: when using defer, all arguments to the function are evaluated immediately, but the function call itself is delayed until the surrounding function returns. So this won’t work:

// Wrong! Shutdown runs immediately
defer LogOnError(s.Shutdown(ctx))

The Shutdown is called immediately, and only the LogOnError call is deferred. That’s not what we want! We want both the shutdown and its error handling to happen when the function returns.

Here’s a cleaner solution using a new interface:

type ShutdownWithError interface {
 Shutdown(ctx context.Context) error
}

func ShutdownWithError(ctx context.Context, s ShutdownWithError) {
 LogOnError(s.Shutdown(ctx))
}

Now we can write:

defer ShutdownWithError(ctx, s)
LogOnError(s.ListenAndServe())

This gives us exactly what we want: proper shutdown timing and error handling. The trick here is that Go’s http.Server automatically implements our ShutdownWithError interface, even though its authors never knew about it. This is one of Go’s powerful features — we can declare new interfaces that existing types automatically implement.

Improving Database Transaction Usage

This pattern is particularly useful for database transactions. Best practices dictate that you should always call Rollback on transactions — it’s either a no-op (when the transaction is already committed) or it prevents resource leaks. The typical pattern for transaction code is:

tx, err := db.BeginTx(ctx, nil)
if err != nil {
 return err
}
// This returns an error we're ignoring!
defer tx.Rollback()

// ... do database stuff using the transaction...
tx.Commit()

But Rollback returns an error, and we’re ignoring it! We can apply the same pattern as before:

type RollbackWithError interface {
 Rollback() error
}

func RollbackWithError(r RollbackWithError) {
 // Ignore ErrTxDone, it just means things were already committed
 LogOnError(r.Rollback(), sql.ErrTxDone)
}

This new function, RollbackWithError, lets us write cleaner rollback code that no longer ignores errors:

tx, err := db.BeginTx(ctx, nil)
if err != nil {
 return err
}
defer RollbackWithError(tx)

// ... do database stuff using the transaction...
tx.Commit()

Use it!

This pattern does more than just make linters happy — it makes your code more reliable. You won’t silently ignore errors, but you also don’t have to change the flow of your code to get that benefit. One interface and one function, and you get the safe behavior you want. While you don’t need to handle every error, you should never be in a situation where errors occur without your knowledge. This pattern ensures you’ll know when things go wrong, even when the errors aren’t otherwise handled.

If you liked this, I recommend also checking out LogOnError and Must, which are related techniques that make it easier to safely handle errors succinctly. All code in this blog post is CC0, which means you can freely use it however you want.

Happy coding!

Highly reliable Go code - Log on error

peter@triplepat.com (Peter Boothe) — Mon, 27 Jan 2025 00:00:00 +0000

If you are writing Go, then you spend a lot of time handling errors or ignoring errors. Ignoring errors is a code smell, but it is sometimes the right thing to do. This post describes a function I’ve found useful for handling less-important errors while keeping both the linter and logging systems happy. It’s almost too small for a blog post, but it’s so useful that I wanted to share it.

The Problem

A lot of Go functions return an error value along with a result. If you know that the error should always be nil, you can use Must to crash when the error is not nil. Sometimes, though, ignoring the error is almost completely fine! This can happen, for example, when you call .Close() on a connection — the close might fail or it might succeed, but either way you are done with the connection.

When the linter tells you to pay attention to an unhandled error that you know isn’t important, you’re in a bit of a bind. Ignoring errors is a code smell, and so is disabling the linter. What can you do?

The Evolution of a Solution

An obvious solution is to log the error. This leads to dozens of lines like this throughout your codebase:

if err != nil {
 slog.Error("error closing connection", "error", err)
}

This works but quickly becomes repetitive and clutters your code, and becomes a testability nightmare. Let’s improve this step by step.

Step 1: Basic Implementation

First we make a simple helper function:

func LogOnError(err error) {
 if err != nil {
 slog.Error("unhandled error", "error", err)
 }
}

This initial version saves us from writing the same code over and over again and provides a central point for error handling. This is usually enough, but some APIs return “normal” errors that shouldn’t be logged. We can improve our function to handle these cases.

Step 2: Adding Exception Handling

Sometimes certain errors are expected and shouldn’t be logged. For example, getting io.EOF from a connection isn’t really an error. We can improve our function to handle these cases:

func LogOnError(err error, exceptions ...error) {
 if err == nil {
 return
 }
 for _, exception := range exceptions {
 if errors.Is(err, exception) {
 return
 }
 }
 slog.Error("unhandled error", "error", err)
}

Now we can explicitly specify which errors we expect and want to ignore:

rtx.LogOnError(os.Remove(filename), os.ErrNotExist)

This removes the file if it exists, and doesn’t log anything if it doesn’t. If the remove fails for some other reason, then a log message is produced.

Step 3: Adding Stack Traces

As a final improvement, we can make debugging easier by logging the source of the error:

func LogOnError(err error, exceptions ...error) {
 if err == nil {
 return
 }
 for _, exception := range exceptions {
 if errors.Is(err, exception) {
 return
 }
 }
 // If we get here, we have an unhandled error.
 _, file, line, ok := runtime.Caller(1)
 if ok {
 slog.Error(
 "unhandled error",
 "error", err.Error(),
 "file", file,
 "line", line,
 )
 } else {
 slog.Error(
 "unhandled error (no debug information)",
 "error", err.Error(),
 )
 }
}

Now we’ll know exactly where each unhandled error originates, making debugging much more straightforward.

Real-World Examples

Here are some common scenarios where LogOnError shines:

HTTP Response Writing

// Encoding JSON - errors here usually mean the client disconnected
rtx.LogOnError(encoder.Encode(response))

File Operations

// Closing files after we're done with them
rtx.LogOnError(file.Close())

// Removing temporary files - don't worry if they're already gone
rtx.LogOnError(os.Remove(tempFile), os.ErrNotExist)

Network Operations

// Closing network connections
rtx.LogOnError(conn.Close())

Database Operations

// Closing database transactions
rtx.LogOnError(tx.Rollback())

Benefits

Using LogOnError provides several advantages:

Cleaner Code: Reduces boilerplate while maintaining proper error handling
Better Observability: All unexpected errors are logged with their location
Centralized Control: Single point to modify error handling behavior
Linter Friendly: Satisfies the linter’s requirements for error handling
Flexible: Easy to customize for different types of errors or logging needs

Use it!

This pattern has become an essential part of my Go toolkit. The beauty of this approach is that it’s simple enough to implement in minutes, yet powerful enough to significantly improve your application’s readability. Handling errors can be like “eating your vegetables” — it’s good for you, but sometimes you don’t want to. This makes that handling much easier.

All code in this post is available with a CC:0 license. You can use it in your own projects and modify it to fit your needs. No credit needed, but I’d love to hear from you if you find patterns like this useful. In my experience, reliable systems come from repeatedly applying small improvements like this until the code is so clear that all problems are obvious.

If you found this helpful, you might also enjoy my post about Must and ValueOrDie for easily handling unrecoverable errors. A common thread between these two is that when the mental “speed bump” of adding an if check for errors is removed, Go code becomes easier to read, understand, and write. It kind of doesn’t matter that the check is trivial, it still takes up brain space, and reclaiming that brain space is a good thing.

This isn’t big and it isn’t rocket science, but it is a small improvement that can make a big difference when used consistently, and it saves you from writing a lot of boilerplate code. Use it in good health, and happy coding!

Highly reliable Go code - Must and ValueOrDie

peter@triplepat.com (Peter Boothe) — Mon, 20 Jan 2025 00:00:00 +0000

With Must and ValueOrDie, you can make error handling in Go a lot easier, clearer, and safer. Even better, this isn’t a new idea; it’s been around for a long time and is used in many places. You should use it too!

When writing Go code, a lot of methods can return an error value. Best practices dictate that you not ignore this error (or any other return value), but instead do one thing if the function succeeded and another if it returned a non-nil error. Because of this, Go code has sometimes been accused of just being if err != nil { ... } over and over again.

Sometimes, however, there is nothing that can be done when there is an error. Your program can’t safely go on, and there’s no way of fixing the problem. Other times, the error is actually impossible, but the compiler can’t prove that. For both cases, people have repeatedly come up with the same solution, and have even named it the same thing: Must().

The first instance I ever saw was MustCompile in the regexp library. The regexp compiler returns a compiled regular expression, and an error in case compilation fails. But sometimes you are compiling a static string! In that case, if compilation ever succeeds it will always succeed, and if it fails your code is wrong and the program should crash.

`Must` and its sibling `ValueOrDie`

The regexp library authors made regexp.MustCompile() for crashing when a regexp failed to compile. There are also other MustX() functions sprinkled around the standard library and in other peoples’ code. The name was evocative, and ever since, people have also proposed making a generic Must() function and put their own Must() functions as helpers. There’s lots of versions with slight variations, but the simplest one is easiest to understand:

func Must(err error) {
 if err != nil {
 panic(err)
 }
}

Thanks to Go generics, we can add its twin

func ValueOrDie[T any](val T, err error) T {
 if err != nil {
 panic(err)
 }
 return val
}

With these two functions, setting up your systems in main() is a lot easier! Do you need to successfully open a file?

 file = ValueOrDie(os.Open(filename))

does the job! Do you need to open a server on a port for serving a service?

 listener := ValueOrDie(net.Listen("tcp", address))

works great! Do you need a string to unmarshal as json successfully?

 Must(json.Unmarshal(scanner.Bytes(), &received))

to the rescue! In each of these cases, the error is fatal, and the program should crash. The Must() and ValueOrDie() functions make this easy.

Functions as folklore

These functions are so useful that they keep getting written as an internal tool in libraries. Most recently I found a Must() in the uuid library. It only works on UUIDs, but it’s defined exactly like ValueOrDie and used in many important and popular library functions (e.g. New()).

Because this pair of little functions is repeatedly and similarly defined in many places, I would encourage you to put them in a standard place in your own projects and use them liberally. They’ve passed the test of time! They also really help when targeting high code coverage, because useless error code paths are covered by the testing of Must() rather than your tests being required to invoke the error condition. You know for sure that if your program got to a particular line, then the previous commands must have succeeded.

When I’m feeling whimsical, I think of these functions as the anthropic principle applied to code. They mean that your program ran successfully at the end because it got to the end, which means it ran successfully! Whimsy aside, this also means that your program is abiding by crash-on-error principles in some important parts, and crash-on-error programs have a history of being better in many contexts because they don’t try and stumble forward when things are broken; they immediately crash. The Pragmatic Programmer agrees! Tip 38 is “Crash Early. A dead program normally does a lot less damage than a crippled one.”

I highly recommend using a linter to make sure you never ignore a returned error, and then that you use Must and ValueOrDie throughout for errors that are impossible or unrecoverable. You save yourself the trouble of testing those error conditions, and people reading your code don’t have to worry that something might not have worked. It also follows the next Pragmatic Programmer tip! Tip 39: “Use Assertions to Prevent the Impossible. If it can’t happen, use assertions to ensure that it won’t. Assertions validate your assumptions. Use them to protect your code from an uncertain world.”

When advice from 20 years ago is being actively used throughout many modern codebases and being constantly re-invented based on need, then I think they were really on to something. If you want your code to be reliable, make sure errors don’t happen. The only way to be sure they don’t is to either handle the errors (which makes them not erroneous, but instead an understood condition) or to crash (which ensures that your program will not erroneously try and make progress). Either way, you need to check the success condition, so turn on that linter and get to it!

Implications for the larger system

The only real requirement after this is a system that notices the crashes and “does the right thing”. In all cases, the number of crashes should be tracked, but sometimes the right thing is to allow the crash to take a larger system out of production, and other times the right thing to do is to restart the crashed program (“Have you tried turning it off and on again?” at scale).

Either way, you can put your system on a better-designed and better-tested path by liberally using Must(...) and ValueOrDie(...). These functions help assert that neither unhandleable errors nor inconceivable conditions are occurring.

Implementing `Must` and `ValueOrDie`

By this point, I think we can agree that these functions are “folklore”. Implemented in many places, often with tiny local variations, but all basically the same. When I wrote open-source Go code, I put these functions in a library called rtx (“run-time extensions”). However, I now think that version is a bit overwrought. It tries too hard to be pretty and clever with its output.

The one I currently use in my projects is:

package rtx

import (
 "os"
 "log/slog"
 "runtime/debug"
)

// We allow injection for testing.
var osExit = os.Exit

func die(err error) {
 debug.PrintStack()
 slog.Error("fatal error", "error", err)
 osExit(1)
}

func Must(err error) {
 if err != nil {
 die(err)
 }
}

func ValueOrDie[T any](obj T, err error) T {
 if err != nil {
 die(err)
 }
 return obj
}

I prefer this version for two reasons:

It calls os.Exit(1), which is unrecoverable, rather than calling panic(), which is recoverable. These functions are for unrecoverable situations. By using an unrecoverable function we remove the temptation to try and use this for fancy flow control.
It prints the stack trace and a simple slog message. I’m a big fan of structured logging and this does that, but it doesn’t get too precious about it by somehow trying to convert a stack trace into a structured log. Also, because we exit instead of panicking, it’s nice to print the stack trace because we don’t get it by default.

You might have slightly different preferences and that’s fine! Each teller tells their own version of a folklore story, and that’s part of the charm.

When to use `Must` and `ValueOrDie`

✅ Good Use Cases:

Program initialization and setup
- Loading configuration files
- Opening required system resources
- Compiling static regular expressions
- Parsing known-valid static data
Test setup where failure means the test itself is broken
Situations where a returned error can not occur
Situations where recovery is impossible or meaningless

❌ Avoid Using For:

Network operations that can be retried
Any runtime data that could legitimately fail
Any time where handling the error is desirable
Errors that don’t matter
Situations where partial failure is acceptable

The key question to ask yourself: “If this fails, does it make sense to continue doing anything?” If not, Must and ValueOrDie are appropriate.

Use it!

I hereby place the code in this blog post in the public domain (CC:0). Please use it for whatever you want in good health, you can even claim it as your own! It’s been written and rewritten so often that at this point it’s folklore and folklore has no owner. Also, if you can figure out where and how to put a version of it somewhere in the Go stdlib, please do that, so people don’t have to keep rewriting it, and we can finally settle on the one true implementation ;)

Happy coding!

Runtime dependencies

peter@triplepat.com (Peter Boothe) — Thu, 09 Jan 2025 00:00:00 +0000

We run a SaaS business for software reliability, which means we want others to depend on us. In order for others to depend on us, we need a demonstrated record of uptime and reliability, and we need to be clear about what we depend on at runtime to deliver that service.

In our keep it boring post, we listed our dependencies. Some of those were services, and some were libraries. For the services, some we depend on to work at least once a month, and some we depend on to work basically all the time. As we get closer to launch it’s a good time to revisit our dependencies and be explicit about them and see if we are happy with the current state of affairs.

Must work all the time: at least one cloud provider

If all of our cloud providers go down in multiple regions simultaneously (that’s 5 machines spread across AWS, GCP, and tiny EU-based TILAA), Triple Pat is down and we can’t do anything about that. That said, it seems vanishingly unlikely that they would all go down simultaneously, and our cockroach-like strategy of having each server be a master DB means that we can survive as long as at least one cloud provider is up.

The critical thing here is that our dependencies on cloud providers is an OR not an AND. If one cloud provider goes down, we are still up. If two of them go down, we are still up. If four regions across three providers go down, we are still up. We need at least one region of at least one cloud provider to be up.

Must work almost all the time: at least one DNS provider

DNS is scary because it’s always DNS and DNS is a critical dependency of the Internet. We use Godaddy as our registrar and our provider for triplepat.com and it’s competitor CloudFlare as our registrar and provider for triplepat.net. Our phone apps will try names from both domains before giving up, and so will all check-ins that follow best practices. If DNS goes down then every system that does not have a cached DNS record for our systems can no longer contact our systems, but users can still contact our systems as long as at least one of our DNS providers is up.

We can survive a few minutes of DNS downtime every now and then thanks to DNS caching. We have two independent names using two different registrars and providers, so it would have to be a pretty major disaster to take both of them down for more than a few minutes.

Must work pretty well: Tailscale

We use Tailscale for our internal network to allow our DBs to sync their records. Tailscale has a good design where even if the Tailscale servers go down for a bit, it is highly likely that our internal traffic will still flow. This means that we can survive periods of Tailscale downtime without worry. Their robustness to their own failure is part of the reason we use them! If they went down for days we would be in trouble, but we don’t expect that to happen. If Tailscale goes down for an extended period, we have a manual failover process that we can use to route around their service.

Must work at least once every 3 months: LetsEncrypt

We use LetsEncrypt to get SSL certificates for our domains. We renew our certificates regularly. If we don’t have a valid certificate, we can’t serve HTTPS traffic. This is a critical dependency for us. That said, the certificates are valid for 3 months, and we renew them at least weekly. So we can survive a few months of LetsEncrypt downtime. If LetsEncrypt disappears, we have months to fix it before our users notice a problem.

That’s all!

In steady state, those are all of our runtime dependencies for check-ins. As a company, we rely on more services that don’t directly affect users’ check-ins:

The Play Store and App Store and their subscription systems allow users to download and use the App andfor us to accept money, which is important for our long-term existence, but not important on a per-check-in basis. If those are down for less than a week, users shouldn’t notice. If they are down for more than a week, it will be in the news and everyone will notice.
Dockerhub stores our images and Github stores our code and runs our CI/CD, but them going down impedes our development process not our runtime systems.
We use Google for Drive, Docs, and Gmail, but those going down inconveniences us but won’t break our service.
We (of course!) monitor all of these services and have alerting for them, so we also rely on Grafana Cloud to alert us when they go down. If Grafana Cloud goes down for a bit, our systems still work for our users.
We use Better Stack for our monitoring our uptime, but if they go down for a bit our systems still work for our users. Ironically, Better Stack was having an outage at time of publication, so our uptime proof was down, a nice example of the intricacies of caring about reliability!

The bottom line is that for our users (you!), as long as at least one cloud provider is working and at least one DNS provider is working, then our service, our app, and your check-ins should all work just fine. Anything else being down might inconvenience us, but you should be fine.

Keep it Boring

peter@triplepat.com (Peter Boothe) — Fri, 03 Jan 2025 00:00:00 +0000

We are selling a service to people who care about reliability, and people who know reliability know that keeping things simple is one of the best steps on the path to keeping things reliable.

So here’s what we use!

We use default, boring technology whenever possible. Every time we are clever, we should know the reason why. We use our cleverness on the core of the service (keeping the check-in service reliable using master-master DBs and CRDTs), and we keep the rest simple.

This means we use:

SQLite for our database.
Tailscale for internal networking.
Letsencrypt for SSL/TLS certificates.
cert-manager to manage them on each host.
nginx for our public web server and reverse proxy.
Docker for containerization.
Docker Hub to hold our container images.
Prometheus to generate our metrics.
Grafana for our monitoring dashboards.
Github for our source code management.
Github Actions for CI/CD.
Go for our server programming language.
Kotlin and Swift for our mobile programming languages.
AWS and GCP for our cloud providers, and only vanilla EC2/GCE machines to avoid lock-in
Godaddy and CloudFlare for our DNS providers
Hugo for our static website generator

We also use every available linter and formatter to make sure our code, configs, and output are internally clean and consistent and align with the expectations of the outside world.

It’s been a great experience so far, and we’re excited to see how it goes.

How the Triple Pat website works

peter@triplepat.com (Peter Boothe) — Thu, 17 Oct 2024 00:00:00 +0000

The website is a static website, built with ~~11ty~~ Hugo, and hosted in Google Cloud. Also running on this server is an instance of the Triple Pat check-in server, and the main instance of the Triple Pat user service. These three systems, as well as the monitoring system, are all behind an nginx web server which takes care of routing requests appropriately.

When building a website, and especially a blog, there’s a real question: do I want to have comments? People love comments (mostly), but it’s hard to add “just a little dynamism” to a website without opening up a huge can of worms. We could have used Disqus (which is a cool service! this is not anti-Disqus, in fact they are the only recommended viable alternative to what we did!), but instead we did something a little funky: we made the comments section be Bluesky threads! Thanks to Aendra Rininsland’s blue-comments library, we can add comments to our website with minimal fuss, and we get notified of new comments as if we were chatting on Bluesky, because we are!

I think it’s a really nice system, because we can have the comments section be 100% Javascript powered, and then Bluesky takes care of user identity, moderation, and all of the challenges that are implicit in social apps:

If we start getting bad actors in the comments, Bluesky’s well-functioning block system means that those comments can be easily removed from view.
We won’t miss a comment, because every comment is a social media notification.
People are in control of their own words. If they want to delete their comments they can do that on Bluesky and the comment will disappear from the website.

We use a social website to empower the social act of commenting, and they then take care of letting people be in control of their own words and identities. We take care of the thing being talked about. It’s a good separation of concerns.

The only downside is that we have to have the blog conversations a little more publicly (Bluesky threads are a little more publicized than a chat in the comments on a blog, but not much more), but the shift in context is not too radical, and I think it’s a good tradeoff for a small blog like this.

Building Triple Pat

peter@triplepat.com (Peter Boothe) — Wed, 16 Oct 2024 00:00:00 +0000

To build a SaaS for software reliability, we need to make sure we are building a system that is, itself, very reliable. Tony Hoare famously said there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies.

To make sure we aren’t adding to the problem, we are repeatedly endeavoring to make the system as simple as possible. Along with simplicity, we want to minimize cleverness. Although it feels strange to mix Harry Potter and the ACM Turing Award, the other principle we hew to when constructing this system is Never trust anything that can think for itself if you can’t see where it keeps its brain. So we want to make our systems as dumb as possible, and when they need to be smart, we want to be extra-clear about where the brain is.

Our check-in system is simplicity itself. It is a service that tracks (UUID, timestamp) pairs, and the only system that is allowed to say what “now” is, is the server. The internal database maintains the invariant that the time is never allowed to decrease, which means that the (UUID, timestamp) pairs form a CRDT, which means that our databases can safely operate with every independent node as a master, and we are guaranteed of eventual consistency.

The only “brain” of note is the phone app. It repeatedly polls Triple Pat servers (any server in our geographically distributed set of servers) for the last check-in time of each of a user’s UUIDs. If the time ever gets too old (configurable in the app), then the app will display an alert. If the Triple Pat servers are down, then the last check-in time will not be updated, so the user will be notified — our alerting system fails noisily.

By keeping the database conceptually simple (although master-master mode is always a little complex) and by putting all the brains in the phone app, we can guarantee that if there are no alerts from Triple Pat, then we know for sure that the last check-in time is within acceptable bounds. If the alarm is firing, then Triple Pat may be broken or the user’s service may be down. Triple Pat continues to exist and have customers only if we make sure that it is almost always the case that the alarm fires when the user’s service is down.

All of this is conceptually simple, but it requires a relatively broad knowledge-base to actually build. We have two servers built in Go (one centralized system for user identity services, and one distributed system that just maintains the (UUID, timestamp) pairs), a website built with 11ty, phone apps built with Kotlin and Swift, and a complicated deployment strategy with servers spread across lots of providers. Each piece is simple and has a well-defined function, which is the only reason it remains tractable.

Triple Pat Blog

Memoryless Scheduling Should Be Your Default

What’s wrong with fixed intervals?

What is memoryless scheduling?

A production-ready implementation

Testing randomness

When do you actually need fixed intervals?

Real-world applications

The industry is getting this wrong

Summary

Use it!

Design of the checkin service

“Making a service that is radically simple and reliable”

Architecture

Defaults

Reliability

Efficiency

Storage

Logging

Monitoring and Metrics

Docker

Health checks

Test coverage

Linting

Error handling and recovery

Command line arguments

Operations

Security

Networking

Website

Time

Memoryless

Subscriptions

Summary

100% test coverage makes your code simpler and better

Software correctness and “The Al Capone theory of sexual harassment”

Tests as intellectual forcing functions

TL;DR.

Learning in public from our incidents

Linting your Go code

Reliable DNS

Highly reliable Go code - use clocks not time

A Practical Example

Key Benefits

Use it!

Reliable Systems via Good Metrics

promlint for linting metrics

How to use promlint

Good metric names

Good metric types

Counters

Histograms

Summaries

Gauges

Summary

Use it!

Highly reliable Go code - Don't ignore errors when you defer, use this pattern instead

Improving http.Server usage

Improving Database Transaction Usage

Use it!

Highly reliable Go code - Log on error

The Problem

The Evolution of a Solution

Step 1: Basic Implementation

Step 2: Adding Exception Handling

Step 3: Adding Stack Traces

Real-World Examples

Benefits

Use it!

Highly reliable Go code - Must and ValueOrDie

Must and its sibling ValueOrDie

Functions as folklore

Implications for the larger system

Implementing Must and ValueOrDie

When to use Must and ValueOrDie

Use it!

Runtime dependencies

Must work all the time: at least one cloud provider

Must work almost all the time: at least one DNS provider

Must work pretty well: Tailscale

`promlint` for linting metrics

How to use `promlint`

Improving `http.Server` usage

`Must` and its sibling `ValueOrDie`

Implementing `Must` and `ValueOrDie`

When to use `Must` and `ValueOrDie`