A green test suite feels reassuring. Every test passed, so the code must be right. But a passing suite only tells you that nothing the tests check is broken. It says nothing about what they never check. So we test the tests, by breaking our own code on purpose.
A test that cannot fail is not a test
Imagine a rule: a person must be sixteen to hold an account. Somebody writes the code and a test for it. The test checks that a thirty-year-old is let in and a ten-year-old is not. It passes.
Now change one character in the rule:
// the rule
const mayHoldAnAccount = (age: number) => age >= 16;
// the same rule, broken on purpose
const mayHoldAnAccount = (age: number) => age > 16;A sixteen-year-old is now refused. Does the test notice? It does not. Thirty is still let in, ten is still turned away, and the suite stays green. The rule is wrong, and nothing told us.
Mutation testing, in plain words
That small, deliberate fault is called a mutant. Mutation testing means making many of them, one at a time, and running the tests against each:
- Change one small thing in the code: flip a comparison, drop a check, swap a value.
- Run the tests.
- If a test fails, the mutant is caught. Good.
- If every test still passes, the mutant survived, and a test is missing. We write it.
The fix for the example is one more test: a sixteen-year-old is let in. With that test in place, the broken rule turns the suite red at once.
How we use it
That order matters. A test we have never seen fail might not be testing anything at all. Seeing it go red first is the proof that it would catch the fault it was written for.
Mutants are not the only check. Every change also runs the full test suite. We feed some code large amounts of random input, to find the cases nobody thought to write down. A verification server re-runs the suites, part of the mutation set and scans for leaked secrets on a schedule, and reports what it finds. At the end of each phase, reviewers try hard to break what we built before we call it done.
What it does not prove
Mutation testing tells you your tests would notice the faults you planted. It cannot prove the code has no faults at all, and it does not replace an independent audit. That is why, on the tracker, anything waiting on an audit cannot show as finished, however many tests it passes.
Why you should care
When a piece of work moves on our tracker, it moved because a repository changed, and that change went through these checks first. Nobody can type a number in. We would rather show you a slower bar that is true than a faster one that is not.