Flaky Test Management

Definition

A flaky test is one that produces different results on identical code — green on one run, red on the next, pass on retry. Flaky test management is the operational discipline around them: measuring flake rate per test and per suite, quarantining offenders fast, assigning owners with a fix deadline, and deleting tests that cannot justify their upkeep. It treats the test suite as production infrastructure, because that is what it is — the quality gate every change passes through.

Why It Matters

Flakiness is corrosive in a way that is easy to miss. One flaky test teaches engineers to click "re-run." Five teach them to assume red means infrastructure. At that point the suite's alarm value is gone — a real regression lands, the build goes red, and everyone waits for it to go green by itself. This is how teams with thousands of tests ship broken checkouts. The flake rate is therefore a leading indicator of delivery risk: when it climbs, trust in the pipeline is already falling, and the real failure is only a matter of time.

How the Discipline Works

1. Measure it — CI records pass/fail per test per run on identical commits; flake rate is a computed metric, not a feeling.
2. Set a budget — a visible target (many teams use under 0.5–1% of runs) with the trend on the engineering dashboard.
3. Detect automatically — retry-once detection flags tests that fail then pass; flagged tests enter the quarantine list with an owner.
4. Quarantine fast — a flaky test moves out of the blocking path within a day, into a quarantined job that still runs and reports.
5. Fix or delete on a deadline — quarantine is a hospital with a discharge date, typically one sprint; unfixed tests get deleted or demoted.
6. Fix the causes, not the symptoms — the recurring offenders: sleeps, time-dependence, shared state, order-dependence, real network and filesystem calls.

Real-World Example

An e-commerce team's checkout suite had drifted to a 4% flake rate. Red builds were routinely re-run twice and merged on the third attempt; "known flaky" was a standing phrase in review comments. Then a genuine payment-routing regression turned the build red, was re-run, went green on an unrelated retry, and shipped — £40,000 in duplicated charges and a painful customer-remediation exercise. The recovery was systematic: flake rate measured per test, a 0.5% budget, automatic quarantine with a one-sprint fix deadline, and a rule that retry counts are reported, never hidden. Three months later the suite ran at 0.3%, red builds were treated as guilty until proven innocent, and the phrase "known flaky" disappeared from the team's vocabulary — because the list it referred to was empty.

Practical Lessons Learned

• Retries hide, they do not heal. Auto-retrying a failed test until it passes converts signal into silence; report the retry rate instead.
• Sleeps are the gateway drug. A fixed sleep(500) is a race condition with a lucky number; wait on conditions, never on clocks.
• Time-bombs ship in January and detonate in February. Tests that depend on wall-clock dates, time zones or locale formats fail on a schedule, not on a change.
• Shared state makes order-dependence. A test that passes alone and fails in the suite is reading something another test wrote — isolate with fresh fixtures, not bigger timeouts.
• Quarantine without a deadline is a graveyard. Every quarantined test needs an owner and a discharge date on the day it is admitted.

Expert Tips

• Run the suite repeatedly on an unchanged commit nightly; anything that flips is flaky by construction, with no debate about whether the code changed.
• Track flake rate per team and per test, and publish the worst offenders — ownership follows visibility remarkably fast.
• Prefer fewer, stronger assertions: a test that checks one behaviour clearly is fixable; a 40-assertion end-to-end script is usually deleted, mourned by no one.
• When a test flakes, capture the environment snapshot — seed, clock, dependency versions — before re-running; flakes debugged from logs after the fact take five times longer.
• Count deletion as a valid fix. A test that asserts nothing meaningful, costs a week a year in re-runs, and flakes monthly is negative value.

Common Mistakes

• Annotating tests with automatic retries as policy, so the suite reports green while the retry metric quietly doubles.
• Quarantining forever — a list of 200 "temporarily" skipped tests that no one has looked at in a year.
• Blaming CI infrastructure for every flake without data; some flakes are the runner, but most are timing and state bugs in the tests themselves.
• Fixing flakes by increasing timeouts, which turns a 10-second suite into a 40-minute one and changes nothing about the race.
• Treating flake fixes as unscheduled charity work, so they always lose to feature tickets until the day the suite's alarm value is gone.

Key Takeaways

• Flaky tests destroy the suite's alarm value; the flake rate is a delivery-risk metric, not an annoyance.
• Measure per test, set a budget, and put the trend where engineering reviews happen.
• Quarantine fast with an owner and a discharge date — hospital, not graveyard.
• Fix causes: condition-based waits, frozen clocks, isolated fixtures, no real network.
• Report retries, never hide them — and count deletion as a legitimate cure.

Related Concepts

Pairs with Continuous Integration, Test-Driven Development, Quality Gate, and Trunk-Based Development.