A test that fails every time is annoying. A test that fails some of the time is corrosive. The always-failing test blocks a pull request and gets fixed within the hour. The flaky test fails in CI, passes on retry, and quietly teaches everyone in the team that a red build doesn’t necessarily mean anything is wrong.
That learned indifference is the real cost. Once a squad internalizes “oh, that one just flakes,” every red build gets a retry instead of a reading, and the genuinely broken commit slides through underneath the noise. Add the compounding tax — bisects that take three runs per step, CI minutes burned on reruns — and a handful of flaky tests can cost more engineering time than the features they’re supposed to protect.
The fix isn’t willpower; it’s a workflow. Make flakes reproducible on demand, treat retries as a bridge rather than a destination, quarantine what you can’t fix today with a forced expiry date, and then actually fix the root causes. This post walks through that workflow with pytest, with a short detour into Go’s built-in equivalents at the end.
What Actually Makes a Test Flaky
Almost every flaky test reduces to the same shape: the test reads inputs it doesn’t fully control. The usual suspects:
- Reading the wall clock twice and comparing the results
- Depending on test execution order or shared module state
- Sharing mutable fixtures, database rows, temp files, or environment variables across tests
- Racing real concurrency with fixed
time.sleep()pauses - Talking to real external services with variable latency and availability
- Seeding randomness — directly, or through a library that does it for you
The common thread is hidden state. A deterministic test computes a function of its declared inputs. A flaky test computes a function of its declared inputs plus the phase of the moon. The debugging strategy is therefore always the same: surface the hidden inputs until the failure obeys a command.
First, Make It Reproducible
The most demoralizing class of flake is order-dependent: passes alone, fails in the full suite. Rather than guessing which upstream test polluted your state, let the machine search for you. pytest-randomly shuffles the test order on every run and re-seeds Python’s randomness at the same time, and it prints the seed it used. When a shuffled order fails, you replay it exactly:
# the shuffle seed is printed at the top of every run, e.g. Using --randomly-seed=1553614239
pytest tests/
# replay that exact order to confirm the failure
pytest --randomly-seed=1553614239 tests/
# rerun whatever order failed in the previous run
pytest --randomly-seed=lastOnce the failing order replays on demand, bisect it the cheap way: run the suspect test together with progressively smaller halves of the suite until the polluting neighbor is cornered. The seed does the bookkeeping; you just follow it.
The Retry Trap
pytest-rerunfailures is honest, well-maintained tooling that is routinely used to lie. Configured globally — --reruns 5 in CI — it makes the build green while the defect stays in the code. Every rerun that ends in a pass is a real failure that someone will eventually hit in production, plus a few minutes of CI time everyone pays for on every commit.
Retries earn their keep in two situations: as a temporary bridge while a fix is in flight, and for failures you explicitly recognize and can’t control — a third-party sandbox API that hiccups, an integration environment shared with other teams. In those cases, scope the retry to the cause instead of blanketing the suite:
import pytest
@pytest.mark.flaky(reruns=3, only_rerun=["TimeoutError", "ConnectionError"])
def test_charges_appear_in_ledger(sandbox_stripe):
charge = sandbox_stripe.create_charge(amount_cents=1200)
assert sandbox_stripe.ledger_eventually_contains(charge.id)The precedence rules are marker first, then command line, then the reruns setting in pyproject.toml, with --force-reruns overriding all three for a one-off investigation. By default only the final attempt of a rerun produces a traceback, and reruns show up as R in the short summary — make your team read those. A CI run with a dozen R entries is not a passing build; it’s a backlog.
Quarantine With an Expiry Date
When a flaky test can’t be fixed this week, you have three bad options and one decent one. Deleting it silently shrinks coverage. Leaving it failing trains people to ignore red. Marking it xfail launders it into “expected” forever. The decent option is quarantine: the test stops blocking CI, remains visible as a skip with a reason, and — this is the part most implementations miss — expires. After the expiry date, it runs again, and either it’s fixed or someone makes a fresh, conscious decision to extend the parking ticket.
First, register the marker so pytest doesn’t warn about it:
# pyproject.toml
[tool.pytest.ini_options]
markers = [
"quarantine(reason: str, until: str): temporarily skipped; 'until' is an ISO date, after which the test runs again",
]Then a small collection hook in conftest.py implements the semantics:
# conftest.py
import datetime
import pytest
def pytest_collection_modifyitems(config, items):
today = datetime.date.today()
for item in items:
marker = item.get_closest_marker("quarantine")
if marker is None:
continue
until = datetime.date.fromisoformat(marker.kwargs["until"])
if today <= until:
reason = marker.kwargs.get("reason", "flaky")
item.add_marker(pytest.mark.skip(
reason=f"quarantined ({reason}) until {until}"
))
# past 'until': no skip is added, so the test runs again and must pass
# tests/test_backfill.py
@pytest.mark.quarantine(
reason="fails when a second boundary lands between two clock reads; fix scheduled",
until="2026-09-30",
)
def test_backfill_deduplicates_rows(db_with_backfill):
assert db_with_backfill.duplicate_row_count() == 0Run pytest -r s and quarantined tests appear in the summary with their reasons and dates, so the debt stays on screen instead of in a Slack thread. Two weeks is a workable default window; anything older than a month is usually a deleted test wearing a disguise.
Fixing the Root Causes
The single most common time-based flake is two independent reads of the wall clock compared for equality. Here's the failure in miniature — a session object truncates its creation timestamp to whole seconds, and the test reads the clock again and compares:
from datetime import datetime, timezone
class Session:
def __init__(self):
self.created_at = datetime.now(timezone.utc).replace(microsecond=0)
def test_created_at_is_now():
session = Session()
assert session.created_at == datetime.now(timezone.utc).replace(microsecond=0)The test reads the clock twice. When a second boundary falls between the two reads — a few times per thousand runs, i.e., every couple of CI builds — equality fails. The robust rewrite brackets the read instead of repeating it:
from datetime import datetime, timezone
class Session:
def __init__(self, clock):
self.created_at = clock()
def test_created_at_is_now():
clock = lambda: datetime.now(timezone.utc)
before = clock()
session = Session(clock)
after = clock()
assert before <= session.created_at <= afterThe assertion is now true by construction whenever the code is correct, regardless of when the scheduler pauses the process. Injecting the clock as a callable also opens the door to freezing time entirely in tests that need specific instants.
The second classic is the fixed sleep waiting for something asynchronous to finish. A hard-coded time.sleep(2) is simultaneously too slow (you pay two seconds on every run) and too fragile (it fails the one time the operation takes 2.1 seconds under load). Replace it with a bounded poll against a monotonic deadline:
import time
def wait_for(condition, timeout=5.0, interval=0.05):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if condition():
return True
time.sleep(interval)
return False
def test_order_reaches_shipped_state(client, order_id):
# client is a fixture wrapping the API under test
assert wait_for(
lambda: client.get_order(order_id).status == "shipped",
timeout=10.0,
), "order never reached 'shipped' within 10s"Note time.monotonic() rather than time.time() — wall-clock adjustments and NTP steps have no business inside a timeout, and a monotonic clock is immune to both. The same discipline extends to the other root causes: pytest's tmp_path fixture already gives every test a unique directory, fixtures should hand each test its own rows rather than a shared database, and anything seeded with randomness should accept the seed as a parameter so a failure prints it.
The Same Discipline in Go
Go ships the two levers pytest needed plugins for. -shuffle randomizes test order inside each package (and prints the seed to replay), and -count reruns tests even when they pass — which is exactly what flake-hunting wants, since a test that fails once in twenty runs will surface in a -count=20 loop:
# shuffle order, replay with the printed seed via -shuffle=N
go test -shuffle=on ./...
# hammer one suspicious package
go test -count=20 ./internal/cartThe full flag reference lives in the go command documentation. Quarantine has no built-in equivalent, but t.Skip with a reason string and a date in it gets you the same visibility — the skip text appears in every verbose run, quietly judging you until it's fixed.
Wrapping Up
Flaky tests are a solvable operational problem, not a fact of nature. The workflow holds up under pressure because each step is small: replay failures with seeds, scope retries to recognized causes with only_rerun, quarantine with an expiry date instead of deleting or ignoring, and fix root causes by controlling time and waiting on conditions instead of hoping.
Start measuring it this week. Count the R entries in your last hundred CI runs and divide by a hundred — that's your flake rate, and it's usually worse than anyone estimates. Pick the top offender, reproduce it with --randomly-seed or -count, and fix the hidden input it's reading. Trust in the red build comes back one deleted flake at a time.