Retry failed tests on GitHub Actions

It is GitHub's only built-in retry, and it is manual. From the run page, or from the terminal:

# Re-run only the failed jobs of a run, on the same commit
gh run rerun <run-id> --failed

The re-run uses the same commit (GITHUB_SHA does not change), is allowed up to 30 days after the original run, and a run can be re-run at most 50 times. The unit is the job: every step of every failed job runs again from checkout, including every test that passed the first time.

Nothing in GitHub Actions retries a job on its own. Teams that want that write a second workflow on workflow_run that calls the re-run API when the first one fails, or use a tool that does it for them. It works, and it is the most expensive way to get past one flaky test. We wrote about why that button is not flaky test management in a separate post.

nick-fields/retry replaces a run step with one that runs again on failure or timeout. It is the most popular way to retry a step on GitHub Actions:

name: Tests

on:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements.txt
      - name: Run tests, retry the step
        uses: nick-fields/retry@v4
        with:
          timeout_minutes: 10
          max_attempts: 3
          command: pytest

Here is what it printed in our sandbox, where one test of four fails about 40% of the time (trimmed to the lines that matter):

Attempt 1
FAILED tests/test_calc.py::test_flaky - assert 0.16118732441770278 > 0.4
==================== 1 failed, 3 passed, 1 warning in 0.59s ====================
Warning: Attempt 1 failed. Reason: Child_process exited with error code 1
Attempt 2
========================= 4 passed, 1 warning in 1.90s =========================
Command completed after 2 attempt(s).

Attempt 2 ran all four tests again to get one of them to pass. On a four-test suite that is half a second. On a real suite it is the whole suite, every time any test flakes, and the chance that something else flakes on the second attempt grows with the size of the suite.

The action is good at what it was built for: commands that fail for reasons outside your code, like a package registry timing out or a service container that is slow to start. retry_on: timeout and retry_on_exit_code let you narrow it to those cases. Wrapped around a test runner, it is "Re-run failed jobs" with fewer clicks.

Your test framework is the only thing in the pipeline that knows which test failed, so it is the only place a retry can target one test. In pytest, mark the test you know is flaky:

import random

import pytest


@pytest.mark.flaky(reruns=2)
def test_flaky():
    assert random.random() > 0.4

and install the plugin in the workflow. Nothing else changes:

name: Tests

on:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements.txt pytest-rerunfailures
      - name: Run tests
        run: pytest

pytest prints an R for each rerun. The other three tests ran once:

tests/test_calc.py ...R.                                                 [100%]
========================== 4 passed, 1 rerun in 0.38s ==========================

Every major framework has the same lever. Scope it to the tests you know are flaky rather than the whole suite: a blanket retry gives a real regression two more chances to pass on luck.

A retry that works is invisible. The job is green, the pull request merges, and the only trace that a test failed is an R in a log nobody opens. Six months later the marker list has thirty entries, nobody remembers why half of them are there, and the suite's real failure rate is a mystery.

"Known flaky" also needs someone to know. On GitHub Actions that knowledge lives in whoever last clicked "Re-run failed jobs" on that test, because Actions keeps logs per run and has no notion of a test across runs. A test report action makes each run's failures readable, and still forgets them when the run ends. The retry decision ends up being made from memory, which is how a test goes from flaking 5% of the time to 40% without anyone noticing.

Test Engine is the test layer GitHub Actions is missing. Its plugins run inside your test framework, so they work at the level a retry should: the test.

name: Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      # requirements.txt lists pytest and pytest-mergify
      - run: pip install -r requirements.txt
      - name: Run tests
        env:
          MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
        run: pytest

• Every attempt is recorded. Your framework's own retries count. In our sandbox, the runs where pytest-rerunfailures rescued the test were green on GitHub, and Test Engine recorded each of them with one failed test. The list of known flaky tests stops depending on anyone's memory.
• Pull requests: quarantine instead of retry. A test Test Engine knows is flaky can be quarantined: it still runs and reports, but its failure no longer fails the job. With a detection budget set, a test the pull request adds is re-run to check it is stable before it merges.
• Default branch: test retry. On pushes to main, a failed test is run again inside the job, up to the retry count you set for the repository and within a time budget for the run. Broken tests and quarantined ones are left alone.

pytest, Vitest, Playwright, Jest and RSpec have native plugins (pytest-mergify above); other runners upload JUnit XML with the Mergify CI action. The retry count and budgets are set per repository in the Test Engine settings of the dashboard.

If a retry is the right call for a whole job (a runner that died, an image pull that timed out), that is a CI problem rather than a test problem, and CI auto-retry covers it. GitHub Actions still sees jobs, not tests. Test Engine is the test layer, and the job layer is next.