How do I debug flaky tests and intermittent CI failures?
Rerun the failing test on the same commit. If the result changes, the code did not cause it, so compare a failing run with a passing one, match the difference to a known cause, and start with the test that costs your team the most.
In one paragraph
A rerun tells you a test is flaky and nothing about why, which is how a suite ends up with a test everybody reruns and nobody fixes. Debugging starts once you hold a red run and a green run of the same code, because the difference between them (order, workers, timing, runner) usually names the cause.
Is it flaky, or did my change break it?
Run it again on the same commit. Two results on one commit mean the outcome does not follow from the code. If it fails twice, run it on the parent commit too: a failure there means your change is not the cause, and a pass means it probably is.
One rerun is a weak test of a failure that shows up one run in twenty, so repeat it until you have a failure to look at:
# Go: 100 runs in shuffled order, with the race detector
go test -run '^TestCheckout$' -count=100 -shuffle=on -race ./...
# pytest, with the pytest-repeat plugin
pytest tests/test_checkout.py --count=50
# Playwright
npx playwright test tests/checkout.spec.ts --repeat-each=50What does the failure pattern point to?
| What you see | Usual cause | What to do next |
|---|---|---|
| Passes alone, fails in the full suite | Test order, or state left behind by an earlier test | Shuffle the order, then cut the tests that run before it in half until the failure goes away. |
| Fails only with parallel workers | A shared database, port or temp directory | Run with one worker to confirm, then give each worker its own copy. |
| Fails at certain hours or dates | The clock, a time zone or a month boundary | Freeze the clock in the test and run with the runner's time zone. |
| Times out on CI, passes on a laptop | A fixed sleep or an unawaited call on a slower machine | Wait for a condition instead of a duration, and limit CPU locally to reproduce it. |
| The job dies and no test failed | The runner ran out of memory, or the network or a service dropped | Read the job log around the exit. There is no test to fix here. |
Each framework has its own favorite version of these. The flaky tests guide has breakdowns for pytest, Jest, Playwright, Go and more.
What should I compare between a failing and a passing run?
Open both logs side by side. Start with the random seed if your runner prints one, since replaying it can turn an intermittent failure into a repeatable one. Then check:
- the test order, and which worker ran the test
- the runner type and image, which can change under a floating tag
- dependency versions, if anything installs without a lockfile
- stderr and timing around the failure, including the test that ran just before it
Is it a flaky test or a flaky job?
A job can fail with every test green, and then the test logs will not help you. Test Engine tracks health per test, and mergify tests show '<full test name or glob>' prints a test's health, ratios and last failure from your default branch. In a script, read .tests[].health_status from --json, since the command exits 0 on a flaky test too. CI Insights gives each job a health status from its recent runs, which is where a runner or setup problem shows up.
Which flaky test do I debug first?
The one that fails most often in a required check, because that is the one stopping people from merging. A test that fails weekly in a nightly job can wait. Detecting and quarantining flaky tests shows how to get the full list without collecting it by hand.
Quarantine the test while you work on it. It keeps running and its failures stop blocking merges, so nobody has to rerun around it during the days the fix takes. Rerunning instead gets the build green and fixes nothing.
How do I know the fix worked?
Rerun the repeat command, with more runs than feels necessary. A test that fails one run in twenty still passes 20 runs in a row about a third of the time. At 100 runs, the odds drop below 1%.
Then watch it on your default branch. Test Engine calls a flaky test healthy again only after a streak of clean passes over several days, because a flaky test can pass for a while and fail again. Turn on unquarantine_when_healthy and the test leaves quarantine once it gets there.
FAQ
What is the difference between a flaky test and a broken test?
A flaky test's result does not follow from the code: it passed and failed on the same commit, for example. A broken test fails and nothing points to flakiness, so the code or the test is simply wrong. They need different work, which is why it pays to tell them apart before debugging.
Should I just retry failed tests?
A retry gets one pull request moving and leaves the test as flaky as it was. It is a reasonable way to keep working while someone fixes the test, and a bad place to stop, because the next person hits the same failure.
Why does a test only fail in CI?
CI machines usually have less CPU, a different number of workers, a clean filesystem and a different time zone from your laptop. Any of those can expose a race or a hidden assumption, so run the repeat command inside a CI job when it will not reproduce locally.
Should I delete a flaky test?
If nobody can say what it protects, yes. A test that fails at random and covers something already tested elsewhere costs more than it catches. Otherwise quarantine it while you work on the fix, so the coverage stays and merges keep moving.
Full reference: test health, detection and quarantine.
Last updated: