Flaky PHPUnit tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, PHPUnit reaches Test Engine as a JUnit report: --log-junit junit.xml writes it, and the mergifyio/gha-mergify-ci action uploads it and sets the job's result with quarantine taken into account. PHPUnit writes the report itself, so phpunit.xml stays as it is. Below: the workflow, why PHPUnit's own retry hides exactly what Test Engine needs to see, and the runner details that make PHPUnit suites flake on GitHub Actions.
Setup
Run PHPUnit with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
.github/workflows/ci.yml: Write junit.xml with PHPUnit's --log-junit, then upload it with the Mergify action, which sets the job's result.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
env:
HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
steps:
- uses: actions/checkout@v7
- uses: shivammathur/setup-php@v2
with:
php-version: "8.5"
coverage: none
- run: composer install --no-interaction --no-progress
- name: Run tests
id: tests
continue-on-error: true
run: ./vendor/bin/phpunit --log-junit junit.xml
- name: Mergify CI Upload
if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
uses: mergifyio/gha-mergify-ci@v28
with:
token: ${{ secrets.MERGIFY_TOKEN }}
report_path: junit.xml
test_step_outcome: ${{ steps.tests.outcome }}
- name: Fail when tests failed and no report was uploaded
if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
run: exit 1continue-on-error: trueon the PHPUnit step hands the job's result to the upload step, which leaves out the failures of quarantined tests.test_step_outcomepasses PHPUnit's own outcome along, so a run that crashed but left a clean-looking report still fails the job.- Pull requests from forks and from Dependabot get no
MERGIFY_TOKEN.HAS_MERGIFY_TOKENskips the upload there, and the last step fails the job when PHPUnit failed and nothing was uploaded. Without that step,continue-on-errorwould turn those red runs green. - setup-php's README notes that Xdebug is enabled by default on Ubuntu GitHub Actions images and recommends turning it off for PHP performance.
coverage: nonedisables both Xdebug and PCOV. If you need coverage, collect it in a job of its own. - PHPUnit 13 requires PHP 8.4.1 or later, so
php-versionhas to match the PHPUnit your lock file installs. Testing several PHP versions in a matrix? Every leg reports the sameGITHUB_JOB, so addjob_name: test (${{ matrix.php }})to the upload step. Without it, a test that only fails on one PHP version looks flaky instead of broken. - Running paratest? It takes the same
--log-junitoption. Swap the command in the test step and keep itsid, and the upload step stays as it is.
Full reference: PHPUnit on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs starts the job over from the checkout, composer install included, and runs every test again to find out whether one of them was unlucky. If the second attempt passes, the pull request goes green, and the failure stays in the log of an attempt nobody opens.
PHPUnit 13.3 added a retry of its own, --retry N and the #[Retry] attribute (changelog). It reruns a failing test inside the run and stops at the first success. The console summary still mentions the failed attempt, but the JUnit logger writes a single testcase from the final attempt, and the failed attempts only add to its time. A test saved by a retry reaches Test Engine as a pass. Leave --retry off the job that uploads.
With the upload in place, every attempt sends its report, on pull requests and on the default branch, and each one adds to the test's history. A test that fails on one attempt and passes on another of the same commit is flagged as flaky. Once it is quarantined, the upload step leaves its failure out when it sets the job's result, and the test keeps running and reporting until it is healthy again.
Runners
Why PHPUnit tests flake on GitHub Actions
paratest at the runner's width
paratest starts one process per CPU core unless you pass
--processes. A GitHub-hostedubuntu-latestrunner has 2 vCPUs and 8 GB on a private repository and 4 vCPUs and 16 GB on a public one, so the same suite runs on two workers in the company repository and four in the open source one, with different tests sharing each worker's static state. Workers that share one database collide on top of that. paratest sets aTEST_TOKENthat is different for every running process: build the test database name from it.# Reproduce a failure from a private-repository runner vendor/bin/paratest --processes=2A new random order on every attempt
With
--order-by random(or the same setting inphpunit.xml), each attempt of the job draws a new order, so a test that depends on what ran before it fails on one attempt and passes on the rerun. PHPUnit prints the seed asRandom Seed:at the top of a randomized run. Copy it from the failed attempt's log and replay that order with --random-order-seed before calling the test flaky../vendor/bin/phpunit --order-by random --random-order-seed 1762367891A database container that is started but not ready
A
services:database starts before your steps run, but started is not ready. Without a health check, the migrations or the first test that opens a connection can reach a PostgreSQL that is still initializing, and the failure lands on whichever test PHPUnit runs first. GitHub's own example adds the health check that makes the job wait for the server.services: postgres: image: postgres env: POSTGRES_PASSWORD: postgres options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 ports: - 5432:5432A library tested without its lock file
Without a committed
composer.lock,composer installresolves the dependencies again on every run. A release that lands between two attempts on the same commit makes the second attempt run different code than the first, and the difference reads as a flaky test. Composer's docs recommend committing the lock file for libraries too, and testing against fresh releases in a separatecomposer updatejob. Give that job its own name, so its failures do not land in the history of the job that runs on the lock file.
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which PHPUnit test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the PHPUnit guide for the patterns behind flaky PHPUnit suites on any CI: test isolation, database transactions, and dataProvider non-determinism.
Frequently asked questions
- Why do my PHPUnit tests pass locally but fail on GitHub Actions?
- On GitHub Actions, look first at the runner's 2 vCPUs on a private repository, which changes how paratest spreads tests across workers, and at random order, which draws a new seed on every attempt. A database container that was not ready and dependencies resolved again without a lock file come next. Replay the failed attempt's seed and run paratest --processes=2 locally before you call the test flaky.
- Does PHPUnit's --retry work with Test Engine?
- It runs, but it hides what Test Engine needs. PHPUnit's JUnit report keeps only the final attempt of a retried test, so a test that failed and then passed on retry is uploaded as a pass and never flagged. Drop --retry and #[Retry] from the job that uploads, and let quarantine keep the known flaky tests from blocking merges.
- Why does the PHPUnit step have continue-on-error?
- So quarantine can work. The upload step decides whether the job passes, and it ignores the failures of quarantined tests. If PHPUnit's failing exit code failed the job first, a quarantined test would still turn the pull request red.
- Can I upload a report from paratest?
- Yes. paratest accepts PHPUnit's --log-junit option, so replace the command in the test step, keep the step's id, and leave the upload step unchanged.