Flaky JUnit tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, JUnit results reach Test Engine through the XML reports Maven Surefire already writes, one TEST-*.xml per test class, with nothing to add to the pom. The mergifyio/gha-mergify-ci action uploads the reports of every module and decides the job's result with quarantine taken into account. The setting to watch is Surefire's own retry: with rerunFailingTestsCount on, a test that fails and then passes is uploaded as a pass.
Setup
Run JUnit with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
pom.xml: A plain JUnit 5 project. Surefire runs the tests and writes one JUnit XML report per test class on its own, so nothing Mergify-specific goes in the pom.<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>te-demo-junit</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.release>21</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.junit</groupId>
<artifactId>junit-bom</artifactId>
<version>5.14.4</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.6.1</version>
</plugin>
</plugins>
</build>
</project>.github/workflows/ci.yml: Run Maven with continue-on-error, then upload every module's Surefire reports with the Mergify action, which sets the job's result.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
env:
HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
steps:
- uses: actions/checkout@v7
- uses: actions/setup-java@v6
with:
distribution: temurin
java-version: 21
cache: maven
- name: Run tests
id: tests
continue-on-error: true
run: mvn -B -fae test
- name: Mergify CI Upload
if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
uses: mergifyio/gha-mergify-ci@v28
with:
token: ${{ secrets.MERGIFY_TOKEN }}
report_path: "**/target/surefire-reports/TEST-*.xml"
test_step_outcome: ${{ steps.tests.outcome }}
- name: Fail when tests failed and no report was uploaded
if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
run: exit 1continue-on-error: trueon the Maven step hands the job's result to the upload step, which leaves out failures of quarantined tests.test_step_outcomepasses Maven's own result along, so a build that failed for a reason the reports do not show, such as a test class that does not compile, still fails the job.-faekeeps Maven going after a module's tests fail, so every module runs its tests and writes its reports, and**/target/surefire-reports/TEST-*.xmlpicks them all up. Integration tests on Failsafe? Runmvn -B -fae verifyand upload**/target/failsafe-reports/TEST-*.xmlas well. Keep theTEST-prefix: Failsafe also writes afailsafe-summary.xmlin that directory, which is not a JUnit report, and an upload that includes it fails.- Pull requests from forks and from Dependabot run without
MERGIFY_TOKEN.HAS_MERGIFY_TOKENskips the upload there, and the last step fails the job when tests failed and nothing was uploaded, sincecontinue-on-errorwould otherwise let those failures through. - Testing on several JDKs in a matrix? Every leg reports the same
GITHUB_JOB, so give the upload step ajob_nameper leg, such asjob_name: test (${{ matrix.java }}). Otherwise a test that only fails on Java 17 looks flaky instead of broken. - On Gradle, run
./gradlew test --continueand upload**/build/test-results/**/*.xml. Gradle names tests differently from Maven (addsAnItem()where Maven reportsaddsAnItem), so moving a project from one build tool to the other starts new histories in Test Engine.
Full reference: JUnit on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs starts the job from the top. The cache: maven input of setup-java brings ~/.m2/repository back, so dependencies are not downloaded again, but the build itself starts over: Maven compiles every module and Surefire runs every test class, to retry the one test that failed. The pull request keeps whichever attempt comes back green.
Surefire's own rerunFailingTestsCount retries inside the run, right after a test fails. Its default is 0, which turns it off. We ran a test that fails 40% of the time through the setup above, with -Dsurefire.rerunFailingTestsCount=2 added to the Maven command. It failed, passed on its rerun, and Maven ended with BUILD SUCCESS and a Flakes: 1 warning. The report did keep the failure, as a <flakyFailure> element inside a passing <testcase>, but the upload step counted 4 tests and 0 failures, and Test Engine stored a pass.
So leave rerunFailingTestsCount at 0 on the runs you upload. Every failure then reaches Test Engine, which records each test's result on pull requests and on the default branch, and a test that fails and passes on the same commit is flagged as flaky. Once it is quarantined, the upload step leaves its failure out of the job's result, and the test keeps running and reporting until it is healthy again.
Runners
Why JUnit tests flake on GitHub Actions
JUnit's parallel mode sized by the runner
Run JUnit Jupiter tests in parallel without choosing a strategy, and it uses the dynamic one with a factor of 1: the parallelism equals the number of available processors. That is 2 on a private-repository runner and 4 on a public one, whatever your laptop has. Tests that share a static field or a database row meet in a different order at each width, so a race that never shows on your machine shows on CI. Pin the parallelism to the runner's to reproduce it.
# src/test/resources/junit-platform.properties junit.jupiter.execution.parallel.enabled = true junit.jupiter.execution.parallel.mode.default = concurrent junit.jupiter.execution.parallel.config.strategy = fixed junit.jupiter.execution.parallel.config.fixed.parallelism = 2Test classes in the order the disk lists them
By default, Surefire runs every test class in one JVM: forkCount is 1 and reuseForks is true, so a static field one class sets is still set when the next class starts. Which class comes next depends on
runOrder, whose default isfilesystem, the order in which the class files are listed on disk. Nothing makes that order match between your laptop and an Ubuntu runner, so a class that only passes after another one can start failing on CI.randomshuffles the classes and prints the seed to replay the order; since Surefire 3.6.0 it also shuffles the methods inside each JUnit 5 class.# Shuffle, then replay the order of a failing run with the seed Surefire printed mvn -B test -Dsurefire.runOrder=random mvn -B test -Dsurefire.runOrder=random -Dsurefire.runOrder.random.seed=<seed>A hung test that holds the job for six hours
A test that never returns, stuck on a lock or waiting for a container that never comes up, keeps the job running until GitHub cancels it, after 360 minutes by default. JUnit Jupiter reads a default timeout for every test and lifecycle method from junit-platform.properties, and can dump every thread's stack to standard output when one runs out. The hung test then fails in minutes, the failure reaches Test Engine like any other, and the dump shows where it was stuck. The default thread mode interrupts the test's thread, so code that swallows the interrupt keeps going; Surefire's
forkedProcessTimeoutInSecondsis the backstop that kills the test JVM.# src/test/resources/junit-platform.properties junit.jupiter.execution.timeout.default = 5 m junit.jupiter.execution.timeout.threaddump.enabled = true
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which JUnit test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the JUnit guide for the patterns behind flaky JUnit suites on any CI: method ordering, Surefire forking, and shared static state in Spring tests.
Frequently asked questions
- Why do my JUnit tests pass locally and fail on GitHub Actions?
- Start with the runner and the order. A private repository's runner has 2 vCPUs, so tests you run in parallel go 2 at a time there under JUnit's default strategy, and Surefire runs your test classes one after the other in a single JVM, in whatever order the runner's disk lists them. Reproduce with a fixed parallelism of 2 in junit-platform.properties and mvn test -Dsurefire.runOrder=random, then replay a failing order with the seed Surefire prints.
- Should I use Surefire's rerunFailingTestsCount?
- Not on the runs you upload. A test that fails and then passes on its rerun is written as a passing testcase with a flakyFailure element inside, and the upload counts it as a pass, so Test Engine never sees the failure it needs to flag the test. Leave it at 0 and let quarantine keep the known flaky tests from blocking merges.
- Does this work with Gradle instead of Maven?
- Yes. Run ./gradlew test --continue, so every project runs its tests even after one fails, and upload **/build/test-results/**/*.xml. Gradle names tests differently from Maven (a method name ends in parentheses, for one), so a project that moves from Maven to Gradle starts new test histories in Test Engine.