Skip to content

Flaky Pest tests on GitHub Actions.
Set up, retried right, quarantined.

On GitHub Actions, Pest reaches Test Engine through the JUnit report it inherits from PHPUnit: --log-junit junit.xml writes it, and the mergifyio/gha-mergify-ci action uploads it and sets the job's result with quarantine taken into account. The test step runs Pest with --ci, like the GitHub Actions example in Pest's own docs. Below: the workflow, why the two Pest features that sound like retries keep flaky tests out of sight, and the runner details that make Pest suites flake on GitHub Actions.

By Julien Danjou, Co-founder & CEO of MergifyUpdated

Setup

Run Pest with Test Engine on GitHub Actions

Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:

.github/workflows/ci.yml: Run Pest with --ci and --log-junit, then upload junit.xml with the Mergify action, which sets the job's result.
name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
    steps:
      - uses: actions/checkout@v7
      - uses: shivammathur/setup-php@v2
        with:
          php-version: "8.5"
          coverage: none
      - run: composer install --no-interaction --no-progress
      - name: Run tests
        id: tests
        continue-on-error: true
        run: ./vendor/bin/pest --ci --log-junit junit.xml
      - name: Mergify CI Upload
        if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
        uses: mergifyio/gha-mergify-ci@v28
        with:
          token: ${{ secrets.MERGIFY_TOKEN }}
          report_path: junit.xml
          test_step_outcome: ${{ steps.tests.outcome }}
      - name: Fail when tests failed and no report was uploaded
        if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
        run: exit 1
  • --ci makes Pest ignore ->only() and run the entire suite. Without it, an ->only() committed by mistake shrinks the run to the focused tests, and the report Test Engine receives shrinks with it.
  • continue-on-error: true lets the upload step set the job's result, leaving out the failures of quarantined tests, and test_step_outcome tells it how the Pest step really ended. Fork and Dependabot pull requests get no MERGIFY_TOKEN: HAS_MERGIFY_TOKEN skips the upload there, and the last step keeps a failing run red when nothing was uploaded.
  • Keep --tia off this job. Pest's test impact analysis replays the stored result of tests whose dependencies did not change instead of running them, and its docs say not to add it to the command that runs your suite on CI. A replayed result is not a new execution, so it tells Test Engine nothing about flakiness.
  • Pest 5 requires PHP 8.4 or later. Testing several PHP versions in a matrix? Every leg reports the same GITHUB_JOB, so add job_name: test (${{ matrix.php }}) to the upload step, or a test that only fails on one version looks flaky instead of broken.

Full reference: Pest on docs.mergify.com.

Retries

Re-run failed jobs versus retrying the test

Re-run failed jobs starts the job over: setup-php, composer install, the Playwright install if you run browser tests, then the whole suite. The attempt that passes is the one the pull request shows.

Pest has two options that sound like a retry, and only one of them retries. --retry reorders the suite so the tests that failed last time run first, and every test still runs once. ->flaky() reruns a failing test, up to 3 times by default, with tearDown and setUp in between. Its retry loop catches each failed attempt inside the test and only rethrows when the last one fails, so a test that passes on its second try is a plain pass in the JUnit report. Test Engine never sees it fail.

Without ->flaky(), every failure reaches the report, and every attempt of the job uploads its own, on pull requests and on the default branch. A test that fails on one attempt and passes on another of the same commit is flagged as flaky. Once it is quarantined, the upload step leaves its failure out of the job's result, and the test keeps running and reporting until it is healthy again.

Runners

Why Pest tests flake on GitHub Actions

  1. One process per core, and 2 cores on a private runner

    With --parallel, Pest creates a process for each available CPU core. A private-repository runner has 2, a public one 4, your laptop maybe 10 or more. Tests that share a static property or one database interleave differently at each width, so an ordering bug that never shows on your machine shows on CI. Reproduce with the runner's count.

    ./vendor/bin/pest --parallel --processes=2
  2. Browser tests waiting 5 seconds on 2 vCPUs

    Pest's browser plugin waits 5 seconds before timing out, and the CI example runs browser tests with --parallel. On a 2-vCPU private runner the browsers, your app and PHP share two cores, and the slowest page of the day misses the wait. Raise the timeout in tests/Pest.php, and run with --trace so the trace of a failed test lands in tests/Browser/Traces, which you can upload as an artifact instead of losing it with the runner.

    // tests/Pest.php
    pest()->browser()->timeout(10000);
  3. Shards that move with every new file

    --shard=1/5 across matrix legs balances test files using the timings in tests/.pest/shards.json. Files that are not in it yet are spread evenly across shards, so a pull request that adds tests can move others to another shard, and an ordering problem shows up on a pull request that never touched the failing test. Keep the shard number out of job_name: shards run different tests, so they can share one name, and a name per shard would split a test's history every time it moves.

  4. Xdebug left on from the docs example

    Pest's own GitHub Actions example sets coverage: xdebug on setup-php. setup-php's README recommends disabling Xdebug when you do not use it, for PHP performance. On a 2-vCPU runner the slower run is what pushes a browser wait or a time-based assertion over its limit. Use coverage: none in the job that uploads to Test Engine, and collect coverage in a job of its own.

The missing layer

GitHub Actions doesn't know what a test is.

GitHub Actions tells you a job failed. Test Engine tells you which Pest test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the Pest guide for the patterns behind flaky Pest suites on any CI: higher-order tests, parallel execution, and the Laravel testing stack.

Frequently asked questions

Why are my Pest tests flaky only on GitHub Actions?
Look at the runner first. --parallel starts one process per core, which is 2 on a private repository, and browser tests share those 2 vCPUs with a 5-second default wait. If you copied Pest's example workflow, Xdebug is on as well, and with --shard a new file can move your test to another shard. Run ./vendor/bin/pest --parallel --processes=2 locally to reproduce.
Does pest --retry rerun failed tests?
No. It runs the tests that failed last time first, then the rest, and every test runs once. The option that retries is ->flaky(), and its failed attempts never reach the JUnit report, so a test it saves is never flagged.
How does a Pest test show up in Test Engine?
Under the file's class and the test's description: test('flaky draw') in tests/Unit/FlakyTest.php is reported as Tests.Unit.FlakyTest.flaky draw. The description is part of the name, so rewording it makes a new test as far as Test Engine is concerned.

Catch the flaky Pest tests in your GitHub Actions runs.

Mergify merges more than half a million pull requests a month without breaking main.