“Re-run Failed Jobs” Is Not Flaky Test Management

Ask a team on GitHub Actions how they handle flaky tests and the honest answer is a button. Open the run, pick Re-run jobs, click Re-run failed jobs, wait. Most of the time it comes back green and the PR merges.

That button is what most teams mean by “flaky test strategy”. Nobody questions it because it works, and that is why the same test is still flaking a year later.

What does “Re-run failed jobs” actually do?

GitHub’s REST reference describes the operation in one line: “Re-run all of the failed jobs and their dependent jobs in a workflow run using the id of the workflow run.” The docs on re-running workflows add the detail that matters for flakiness: the re-run will “use the same GITHUB_SHA (commit SHA) and GITHUB_REF (git ref)” as the original run. You get up to 30 days to do it, and “A workflow run can be re-run a maximum of 50 times.”

So a rerun works on a whole job, with the exact same commit as input.

That second part matters. Google’s testing team defines a flaky result as “a test that exhibits both a passing and a failing result with the same code.” A job that fails, then passes on a re-run of the same SHA, is that definition playing out on your own CI: the exact evidence you need to call a test flaky. Then everyone moves on and nothing gets written down.

Why does re-running a failed job not fix flaky tests?

Because the button only answers “can this PR merge now?” It never asks which test is lying, or since when.

A job rerun has no test identity. The button operates on jobs. If test_checkout_flow failed inside a 12-minute integration job, the rerun runs all 400 tests in that job again. The run page keeps the previous attempt around if you go looking, but nothing extracts the fact that matters: this test, at this commit, failed then passed.

A job rerun has no memory. Tomorrow the same test fails on someone else’s PR. They don’t know it failed on yours yesterday. They click the same button, it goes green, and the same nothing gets recorded. Multiply that by every engineer and every week, and the team has watched the same flaky test fail dozens of times without anyone naming it.

It is cheap per click and expensive in aggregate. GitHub’s billing docs use the example of a 10-minute workflow that fails after five minutes: “If you fix the problem and re-run the workflow successfully, in total you’ll use 15 minutes of the repository owner’s allowance.” A flaky retry is the same bill with nothing fixed at the end, and you pay for every test in the failed job to re-test one of them. On a busy private repo that adds up, but it is the cheap part.

What is the real cost of rerunning flaky tests?

The real cost is that engineers learn to stop reading red.

Google measured this on its own CI back in 2016. About 1.5% of all test runs reported a flaky result, and “about 84% of the transitions we observe from pass to fail involve a flaky test!” Your ratio will differ, but when most pass-to-fail flips are noise, people adapt in the obvious way. The same post puts it plainly: “It is human nature to ignore alarms when there is a history of false signals coming from a system.”

Rerun-as-strategy turns that adaptation into muscle memory. A real bug that fails 30% of the time (a race in your code, a test polluted by the one before it) passes on the first rerun 70% of the time, and within two reruns 91% of the time. The engineer who clicked has no way to tell it apart from the flaky test they reran yesterday, so it merges. We covered the test-level version of this in why jest.retryTimes() hides bugs. The button is the same problem one level up, applied by hand.

Google tried automating its way out with automatic reruns and a flag that only reports a failure after three in a row. The verdict, in the same post: it “encourages developers to ignore flakiness in their own tests unless their tests start failing 3 times in a row, which is hardly a perfect solution.”

Is retrying a failed CI job ever the right call?

Yes. A runner gets preempted, or the package registry returns a 502 halfway through an install. None of that is your code, so a retry is the right call.

That is the line worth drawing. A retry is a mitigation: it gets this PR unstuck. Management is what you do with the information the retry just produced. A team that only retries pays for its flaky tests forever, a few minutes at a time.

What does flaky test management actually look like?

It starts with a definition you can compute. “Same commit, different outcomes” is the one Google uses, and it is the one that survives contact with real CI: no human judgement, no guessing from error messages. Every same-commit rerun is a data point for it, as long as something records the outcome per test.

Then you need history at the test level. A test that flaked once in 2,000 runs and a test that flakes on every third PR are different problems. You only find out which is which by recording every outcome of every test against the commit it ran on, for weeks, on the branch that matters.

That history lets you separate flaky from broken. A broken test fails every time and needs a fix or a revert today. A flaky test needs an owner and an investigation. History cannot tell a flaky test from a real race in your code, since both pass and fail on the same commit, and no tool makes that call for you. What history does is put a name and a failure rate on it: a test that fails on 30% of runs sits at the top of a list instead of being rerun past one PR at a time.

Next is quarantine that keeps the test running. Skipping a flaky test (@pytest.mark.skip, xit) throws away the signal you need to fix it and to know when it’s fixed. A quarantined test still executes and still reports; its failures just stop blocking merges. Once someone fixes it and the history shows it healthy, it comes back. Quarantine buys time for that fix, and a test failing a third of its runs still needs a person to look at it. More on the mechanics in our test quarantine guide.

The last piece is the one almost everyone skips: stop new flakiness at the PR. Every flaky test on main was once a new test on a pull request that passed once and got merged. Rerunning the tests a PR adds, before it merges, is the cheapest place to catch it, because the author still has the context and nobody else has been burned yet.

How Mergify Test Engine handles flaky tests

Test Engine is native in every test framework and does detection and remediation from inside it, on any CI: GitHub Actions, Buildkite, Jenkins, or anything else through the Mergify CLI.

A known flaky test is retried in the job. When a test Test Engine already knows is flaky fails, it is retried right where it failed, inside the framework, and the job goes green without anyone clicking rerun. Only known flaky tests get that second try, so a real bug does not pass on a retry. That is the button done right: one test, on the same commit, with the result recorded.

Prevention reruns the tests a pull request adds or changes, on the same commit, and blocks the merge if one of them turns out flaky. Detection marks every test healthy, flaky or broken from its results across CI runs, and ranks the unhealthy ones by impact.

Quarantine runs itself. A test is quarantined automatically once the evidence holds: it has been unhealthy for a while, and the run that flagged it did not turn many tests red at once, so an infra hiccup does not quarantine half the suite. It is released automatically once it is healthy again. In the docs’ words, “A quarantined test still executes in your CI pipeline and its results are still collected by Mergify.” Its failures just stop blocking merges, on any branch.

Merge queue runs are ordinary CI runs, so the same retries and quarantine apply there, and a flaky test never throws a pull request out of the queue. Coding agents get the same data: the Mergify CLI tells your agent each test’s health and quarantine state, so it fixes the right tests.

Keep the button, it’s fine when a runner dies. But the next time a re-run goes green on the same SHA, you are holding evidence that something in that job is flaky. The only question is whether anything records which test it was.