---
title: "Flaky pytest tests on GitHub Actions: setup, retries and runner pitfalls"
description: "Run pytest with Mergify Test Engine on GitHub Actions: the workflow, why Re-run failed jobs hides flaky tests, and what makes pytest flake on GitHub-hosted runners."
url: https://mergify.com/learn/flaky-tests/github-actions/pytest
---

[Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/pytest

# Flaky pytest tests on GitHub Actions.  
Set up, retried right, quarantined.

On GitHub Actions, pytest needs one package and one secret: install `pytest-mergify` and give the step that runs pytest a `MERGIFY_TOKEN`. The plugin uploads every test result when the session ends, applies quarantine inside the session, and reruns new tests before the pull request can merge. Below: the workflow, how that differs from rerunning the job, and the runner details that make pytest suites flake on GitHub Actions.

By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated October 28, 2026

Setup

## Run pytest with Test Engine on GitHub Actions

Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then:

`requirements.txt`: Add pytest-mergify next to pytest in the requirements your tests install.

```
pytest
pytest-mergify
```

`.github/workflows/ci.yml`: Pass the MERGIFY\_TOKEN secret to the step that runs pytest. Nothing else changes in the workflow.

```
name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - run: pip install -r requirements.txt
      - name: Run tests
        env:
          MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
        run: pytest
```

-   No upload step and no `continue-on-error`. Quarantine is already reflected in pytest's exit code, so the step's result is the job's result.
-   Running pytest across a matrix? Every leg reports the same `GITHUB_JOB`, so set `MERGIFY_TEST_JOB_NAME: test (${{ matrix.python-version }})` on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken.
-   The plugin ships CPython wheels only. If a matrix leg runs PyPy, add it as `pytest-mergify; platform_python_implementation == "CPython"` so that leg skips it.
-   Tests run through tox? tox does not pass the job's environment through. Add `CI`, `MERGIFY_*`, `GITHUB_*` and `RUNNER_*` to `pass_env`, or the plugin cannot see the token or the run it belongs to.

Full reference: [pytest on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/pytest).

Retries

## Re-run failed jobs versus retrying the test

GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed.

When a pytest test fails on GitHub Actions, the button you get is Re-run failed jobs. It runs the job again from the top: checkout, setup-python, the dependency install, then the whole session, to find out whether one test was unlucky. If the second attempt passes, the pull request turns green, and the failure is only visible to whoever opens the first attempt's log.

`pytest-rerunfailures` moves the retry inside the job, which is cheaper, but `--reruns 2` applies to every failing test and reports the last attempt. A test that fails one run in three is green in CI and still broken.

pytest-mergify reruns inside the session too, with a narrower rule. On a pull request it reruns the tests the target branch does not have yet, as many times as the Prevention budget allows, and fails the run if any attempt fails, so a new flaky test cannot merge. On pushes to the default branch it reruns the tests Test Engine already knows as unhealthy, to keep their status current, without changing the result. Between reruns it holds back the fixture finalizers pytest would run, so a module-scoped fixture is not torn down mid-rerun.

Runners

## Why pytest tests flake on GitHub Actions

1.  ### Two cores on private repositories
    
    [GitHub-hosted](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) `ubuntu-latest` has 4 vCPUs and 16 GB on public repositories, and 2 vCPUs and 8 GB on private ones. `pytest -n auto` starts one xdist worker per CPU, so the same suite runs on four workers in the open source project and on two in the company repository, with a different split of tests per worker. Timeouts tuned on a ten-core laptop fail first. Reproduce with the worker count of the runner that failed, not your machine's.
    
    ```
    # Reproduce a failure from a private-repo runner
    pytest -n 2 -p no:randomly
    ```
    
2.  ### Service containers that are started but not ready
    
    A `services:` database starts before your steps run, but started is not ready. Without a health check, the first tests can reach a PostgreSQL that is still initializing, and the failures land on whichever tests pytest happens to run first. [GitHub's own example](https://docs.github.com/en/actions/tutorials/use-containerized-services/create-postgresql-service-containers) adds the health check; keep it.
    
    ```
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_PASSWORD: postgres
        ports: ["5432:5432"]
        options: >-
          --health-cmd pg_isready
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10
    ```
    
3.  ### A rerun that installs different packages
    
    setup-python's `cache: pip` caches downloads, not your environment, and an unpinned requirements file is resolved again on every run. A dependency released between two attempts on the same commit makes the second attempt run different code than the first, and the difference reads as a flaky test. Install from a lock file (`uv.lock`, `poetry.lock`, pip-compile output) so a rerun of the same SHA installs the same packages.
    
4.  ### Runners in UTC
    
    Hosted runners run in UTC. A test that builds "today" from `datetime.now()` and compares it with a date computed elsewhere passes all day in Paris or San Francisco and fails on CI in the hour around midnight UTC. Freeze the clock in the test (`time-machine`, `freezegun`) instead of trusting the runner's.
    

The missing layer

## GitHub Actions doesn't know what a test is.

GitHub Actions tells you a job failed. Test Engine tells you which pytest test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [pytest guide](https://mergify.com/learn/flaky-tests/pytest) for the patterns behind flaky pytest suites on any CI: fixture teardown races, xdist ordering, and Hypothesis seed non-determinism.

## Other frameworks on GitHub Actions

-   [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest)
-   [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright)
-   [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec)
-   [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest)
-   [Go](https://mergify.com/learn/flaky-tests/github-actions/go)
-   [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress)
-   [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit)
-   [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng)
-   [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest)
-   [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest)
-   [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit)
-   [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest)
-   [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit)
-   [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust)

## Frequently asked questions

Why do my pytest tests pass locally but fail on GitHub Actions?

The usual suspects on GitHub Actions: fewer CPUs than your laptop (2 on a private repository), which changes how pytest-xdist spreads the tests; a service container that was not ready yet; dependencies resolved again on every run; and a runner clock in UTC. Reproduce with the runner's worker count and the same lock file before you call it flaky.

Do I need to upload a JUnit XML file from pytest?

No. pytest-mergify uploads the results itself when the session ends, so the workflow has no upload step and no continue-on-error. If another tool still reads a JUnit file, keep --junitxml for it.

Does pytest-mergify slow down my GitHub Actions job?

The upload happens once, at the end of the session. Reruns only happen within the budgets you set: on a pull request for tests that are new, on the default branch for tests already known as unhealthy. That is a few tests, not the session, which is what Re-run failed jobs costs.

## Catch the flaky pytest tests in your GitHub Actions runs.

Mergify merges more than half a million pull requests a month without breaking main.

[Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/pytest)
