100% test coverage makes your code simpler and better

Every line of Go code that runs Triple Pat is tested by some test.

% go test ./... -cover=1 -count=1 \
    | awk '{print $5}' \
	| sort \
	| uniq -c
  41 100.0%

That’s 41 packages, and every line is executed at least once in our automated test suite. Why is this good? Why is this not just a silly exercise in pedantic completionism? There’s two reasons: “the Al Capone theory of sexual harassment” and the value of an intellectual forcing function.

Software correctness and “The Al Capone theory of sexual harassment”

Al Capone went to jail not for being a mobster, but for cheating on his taxes. His dishonesty and criminality in one area implied that he was also criminal in others. “The Al Capone theory of sexual harassment” is that sexist harassers in the workplace are likely to also be dishonest and bad for the business in many ways, and should be gotten rid of ASAP. Code that is bad is also likely to be bad in many ways.

The most basic test is the smoke test, where you “turn it on and see if it catches fire”. In a software context, this means running the code and verifying that it didn’t cause a crash. This is a valuable first step towards correctness, especially for greenfield development, because bad code is often bad in many ways (like Al Capone was!). Even the most basic smoke test or in-out test is enough to uncover very real problems.

As code ages and corner-cases are uncovered, the tests will naturally become more complex. But to start out it’s often enough to just make sure nothing crashes by fully covering the code with the most basic of tests.

Tests as intellectual forcing functions

Let’s start with a quote from Tony Hoare:

The real value of tests is not that they detect bugs in the code, but that they detect inadequacies in the methods, concentration, and skills of those who design and produce the code.

When you pursue 100% coverage you design your code for testability. You also find out that many errors are actually impossible (not just unlikely — impossible). A good example is the library google.golang.org/api/idtoken, which has idtoken.NewValidator(context.Context, ...ClientOption) that returns a value and an error. If you don’t pass in any client options, then it is literally impossible for that function to return an error. Therefore, you can (and should, IMO!) use ValueOrDie on its return pair if you aren’t passing in any options.

I would never have known that this was the case if I hadn’t committed to 100% code coverage - I would have just done a mindless error check that returns (nil, err) to the caller. Indeed, that’s the code I wrote before I started trying to test it!

Because the CI coverage report showed this codepath as untested, I tried to cause the error, and I discovered that the error was impossible. This was actually the only error-producing code in the function, so as a result I could also simplify the function I was writing. This insight allowed me to change the code from:

func NewVerifier(ctx context.Context, clientID string) (Verifier, error) {
	validator, err := idtoken.NewValidator(ctx)
    if err != nil {
        return nil, err
    }

	return &verifier{
		validator: validator,
		clientID:  clientID,
	}, err
}

to:

func NewVerifier(ctx context.Context, clientID string) Verifier {
	// An error is impossible because 
	// we're not using a custom transport.
	validator := rtx.ValueOrDie(idtoken.NewValidator(ctx))

	return &verifier{
		validator: validator,
		clientID:  clientID,
	}
}

Pursuit of 100% test coverage forced me to think more deeply about my errors and how to cause them, which led to me discovering that I could make my code simpler, not just in this function but for its callers as well. The simplicity we found “cascaded upwards”! I would never have made this discovery without being forced to, however, because I am a lazy programmer just like everyone else. It was easy to write that error pass-through, so that’s what I did and that’s what most programmers would do. Without this goal of 100% coverage forcing me to think about things, I would not have thought hard enough to notice the opportunity for simplification.

When you think about your code, your code becomes better. The problem is that thinking is hard and tiring, so you need a reason to do it. I recommend making 100% test coverage one of your reasons.

TL;DR.

Every line of code should be executed at least once by your tests. 100% coverage doesn’t mean “no bugs” of course (and for code that needs to accomplish a task, you should definitely also have some assertions about the desired task being accomplished), but full coverage does mean that all remaining crashes are in some sense sneaky. It means they have to be joint test/code bugs, rather than just bugs in the code.

Even “smoke tests” with no explicit assertions implicitly assert that the code they run doesn’t crash. 100% coverage helps keep your code free of simple bugs.

When you try and make sure every line at least gets a smoke test, then you also end up finding that some errors are actually impossible. Impossible errors represent code that you don’t need to write, and complication you can avoid. 100% coverage helps keep your code simple.

100% coverage helps keep code simple and free of simple bugs. Ultimately, the goal is to write correct code, and as Tony Hoare also said:

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

In our twin pursuits of simplicity and correctness, full test coverage plays a valuable role.