A retried test is a deleted test

A flaky test is one that passes and fails on the same code. The usual response is to add a retry, and the usual result is a test that no longer detects anything.

Consider what a retry actually says: this test fails sometimes, and we have decided in advance not to look. Once it is retried three times, a genuine bug that fails one run in three is invisible. The test is still there, still green, still counted in the coverage report, and it has stopped working.

The taxonomy is short and every entry is fixable. Clock — code that reads the current time and behaves differently at midnight, or on the 29th of February. Randomness — unseeded. Locale and time zone — string comparison and date formatting both change under you. Iteration order — object keys, Set ordering, filesystem listings. Shared fixtures — one test leaves state another test finds. Parallelism — the same, but harder to reproduce.