Flaky pytest tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, pytest needs one package and one secret: install pytest-mergify and give the step that runs pytest a MERGIFY_TOKEN. The plugin uploads every test result when the session ends, applies quarantine inside the session, and reruns new tests before the pull request can merge. Below: the workflow, how that differs from rerunning the job, and the runner details that make pytest suites flake on GitHub Actions.
Setup
Run pytest with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
requirements.txt: Add pytest-mergify next to pytest in the requirements your tests install.pytest
pytest-mergify.github/workflows/ci.yml: Pass the MERGIFY_TOKEN secret to the step that runs pytest. Nothing else changes in the workflow.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.13"
- run: pip install -r requirements.txt
- name: Run tests
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
run: pytest- No upload step and no
continue-on-error. Quarantine is already reflected in pytest's exit code, so the step's result is the job's result. - Running pytest across a matrix? Every leg reports the same
GITHUB_JOB, so setMERGIFY_TEST_JOB_NAME: test (${{ matrix.python-version }})on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken. - The plugin ships CPython wheels only. If a matrix leg runs PyPy, add it as
pytest-mergify; platform_python_implementation == "CPython"so that leg skips it. - Tests run through tox? tox does not pass the job's environment through. Add
CI,MERGIFY_*,GITHUB_*andRUNNER_*topass_env, or the plugin cannot see the token or the run it belongs to.
Full reference: pytest on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
When a pytest test fails on GitHub Actions, the button you get is Re-run failed jobs. It runs the job again from the top: checkout, setup-python, the dependency install, then the whole session, to find out whether one test was unlucky. If the second attempt passes, the pull request turns green, and the failure is only visible to whoever opens the first attempt's log.
pytest-rerunfailures moves the retry inside the job, which is cheaper, but --reruns 2 applies to every failing test and reports the last attempt. A test that fails one run in three is green in CI and still broken.
pytest-mergify reruns inside the session too, with a narrower rule. On a pull request it reruns the tests the target branch does not have yet, as many times as the Prevention budget allows, and fails the run if any attempt fails, so a new flaky test cannot merge. On pushes to the default branch it reruns the tests Test Engine already knows as unhealthy, to keep their status current, without changing the result. Between reruns it holds back the fixture finalizers pytest would run, so a module-scoped fixture is not torn down mid-rerun.
Runners
Why pytest tests flake on GitHub Actions
Two cores on private repositories
GitHub-hosted
ubuntu-latesthas 4 vCPUs and 16 GB on public repositories, and 2 vCPUs and 8 GB on private ones.pytest -n autostarts one xdist worker per CPU, so the same suite runs on four workers in the open source project and on two in the company repository, with a different split of tests per worker. Timeouts tuned on a ten-core laptop fail first. Reproduce with the worker count of the runner that failed, not your machine's.# Reproduce a failure from a private-repo runner pytest -n 2 -p no:randomlyService containers that are started but not ready
A
services:database starts before your steps run, but started is not ready. Without a health check, the first tests can reach a PostgreSQL that is still initializing, and the failures land on whichever tests pytest happens to run first. GitHub's own example adds the health check; keep it.services: postgres: image: postgres:17 env: POSTGRES_PASSWORD: postgres ports: ["5432:5432"] options: >- --health-cmd pg_isready --health-interval 5s --health-timeout 5s --health-retries 10A rerun that installs different packages
setup-python's
cache: pipcaches downloads, not your environment, and an unpinned requirements file is resolved again on every run. A dependency released between two attempts on the same commit makes the second attempt run different code than the first, and the difference reads as a flaky test. Install from a lock file (uv.lock,poetry.lock, pip-compile output) so a rerun of the same SHA installs the same packages.Runners in UTC
Hosted runners run in UTC. A test that builds "today" from
datetime.now()and compares it with a date computed elsewhere passes all day in Paris or San Francisco and fails on CI in the hour around midnight UTC. Freeze the clock in the test (time-machine,freezegun) instead of trusting the runner's.
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which pytest test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the pytest guide for the patterns behind flaky pytest suites on any CI: fixture teardown races, xdist ordering, and Hypothesis seed non-determinism.
Frequently asked questions
- Why do my pytest tests pass locally but fail on GitHub Actions?
- The usual suspects on GitHub Actions: fewer CPUs than your laptop (2 on a private repository), which changes how pytest-xdist spreads the tests; a service container that was not ready yet; dependencies resolved again on every run; and a runner clock in UTC. Reproduce with the runner's worker count and the same lock file before you call it flaky.
- Do I need to upload a JUnit XML file from pytest?
- No. pytest-mergify uploads the results itself when the session ends, so the workflow has no upload step and no continue-on-error. If another tool still reads a JUnit file, keep --junitxml for it.
- Does pytest-mergify slow down my GitHub Actions job?
- The upload happens once, at the end of the session. Reruns only happen within the budgets you set: on a pull request for tests that are new, on the default branch for tests already known as unhealthy. That is a few tests, not the session, which is what Re-run failed jobs costs.