Flaky TestNG tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, TestNG results reach Test Engine as the JUnit XML report Maven Surefire writes for the run. The pom below carries the Surefire configuration from the Mergify docs, and the mergifyio/gha-mergify-ci action uploads the report and decides the job's result with quarantine taken into account. Two details decide what Test Engine sees: which file the upload pattern picks up, and what a retry analyzer did to a failure before it was written down.
Setup
Run TestNG with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
pom.xml: Pin Surefire and set TestNG's listeners as the Mergify docs do. Surefire writes the JUnit XML report that the upload step reads.<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>te-demo-testng</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.release>21</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>7.12.0</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.5.3</version>
<configuration>
<properties>
<property>
<name>usedefaultlisteners</name>
<value>false</value>
</property>
<property>
<name>listener</name>
<value>org.testng.reporters.JUnitReportReporter</value>
</property>
</properties>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
</configuration>
</plugin>
</plugins>
</build>
</project>.github/workflows/ci.yml: Run Maven with continue-on-error, then upload Surefire's report with the Mergify action, which sets the job's result.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
env:
HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
steps:
- uses: actions/checkout@v7
- uses: actions/setup-java@v6
with:
distribution: temurin
java-version: 21
cache: maven
- name: Run tests
id: tests
continue-on-error: true
run: mvn test
- name: Mergify CI Upload
if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
uses: mergifyio/gha-mergify-ci@v28
with:
token: ${{ secrets.MERGIFY_TOKEN }}
report_path: target/surefire-reports/*.xml
test_step_outcome: ${{ steps.tests.outcome }}
- name: Fail when tests failed and no report was uploaded
if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
run: exit 1- The file the upload step sends is Surefire's own
target/surefire-reports/TEST-TestSuite.xml: every test of the run in one report, each named by class and method, such ascom.example.FlakyTest.sometimesFails. TheJUnitReportReporterlistener writes per-class copies one level down, injunitreports/, whichtarget/surefire-reports/*.xmldoes not reach. Keep the pattern that shallow: those copies describe the same tests, and a recursive pattern would send each one twice.usedefaultlistenersset to false turns off the HTML and XML reports TestNG attaches by default; on our runs, Surefire's report was the only file the pattern matched. continue-on-error: trueon the Maven step hands the job's result to the upload step, which leaves out failures of quarantined tests.test_step_outcomepasses Maven's own result along, so a build that failed for a reason the report does not show, such as a test class that does not compile, still fails the job.- Pull requests from forks and from Dependabot run without
MERGIFY_TOKEN.HAS_MERGIFY_TOKENskips the upload there, and the last step fails the job when tests failed and nothing was uploaded, sincecontinue-on-errorwould otherwise let those failures through. - The pom pins Surefire 3.5.3, as the Mergify docs do, and that version runs TestNG through its own TestNG provider. Starting with Surefire 3.6.0, TestNG runs through the TestNG JUnit Platform engine instead. If your build already uses 3.6 or later, look at the file names in
target/surefire-reports/after one run and pointreport_pathat theTEST-*.xmlfiles you find there. - On Gradle, call
useTestNG()in thetesttask, run./gradlew testand uploadbuild/test-results/test/*.xml. Gradle writes those JUnit XML reports itself, with no listener to configure.
Full reference: TestNG on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs starts the job from the top. The cache: maven input of setup-java brings ~/.m2/repository back, but Maven compiles the tests again and Surefire runs the whole suite to retry the one method that failed. The pull request keeps whichever attempt comes back green.
TestNG retries inside the run when a test has a retry analyzer: after each failure, it asks the IRetryAnalyzer bound with @Test(retryAnalyzer = ...) whether to run the test again. We bound one that allows two retries to a test that fails 40% of the time and ran it through the setup above. The test failed, passed on its retry, and Maven ended with BUILD SUCCESS. Surefire's report listed the test twice: the failed attempt as <skipped>, carrying the assertion message, and the retry as a plain pass. Test Engine stored one skip and one pass for that run, and no failure.
Test Engine flags a test as flaky when it fails and passes on the same commit, and a skip is not a failure. So take the retry analyzer off the runs you upload. Every failure then reaches Test Engine, which records each test's result on pull requests and on the default branch. Once a flaky test is quarantined, the upload step leaves its failure out of the job's result, and the test keeps running and reporting until it is healthy again.
Runners
Why TestNG tests flake on GitHub Actions
Five threads on two vCPUs
TestNG's thread count does not follow the machine. Set
paralleland Surefire runs your tests on 5 threads unless you changethreadCount. On a private-repository runner, those 5 threads share 2 vCPUs, where a public repository's runner has 4 and your laptop likely more. A method that gives another thread a fixed delay to finish its work gets less CPU inside that delay than it does locally, and fails on CI only. Classes whose methods share fields can opt out of method-level parallelism:singleThreaded = truekeeps all of a class's methods on one thread, even underparallel="methods".@Test(singleThreaded = true) public class CartTest { // every test method of this class runs on the same thread }Test JVMs counted from the runner's cores
With
forkCount=1C, Surefire multiplies the number by the CPU cores: 2 test JVMs at once on a private-repository runner, 4 on a public one, one per core on your laptop. WithreuseForkson by default, test classes are handed to the forks one by one, so which classes share a JVM, and with it every static field, changes with the core count. A class that only passes because another one warmed a cache before it can fail on CI first. The forks also run at the same time, so give each one its own database schema or port with${surefire.forkNumber}, and reproduce with the runner's count,mvn test -DforkCount=2.<configuration> <forkCount>1C</forkCount> <systemPropertyVariables> <databaseSchema>test_${surefire.forkNumber}</databaseSchema> </systemPropertyVariables> </configuration>A hung test that holds the job for six hours
A test that never returns keeps the job running until GitHub cancels it, after 360 minutes by default. TestNG's
timeOutfails a method that runs past its limit, in parallel and sequential runs alike, but only on the methods that carry one. Surefire'sforkedProcessTimeoutInSeconds,surefire.timeouton the command line, covers the whole run: Surefire kills the test JVM after that many seconds, and the job fails in minutes instead of hours. Set it well above your slowest green run, or a busy runner trips it.- name: Run tests id: tests continue-on-error: true run: mvn test -Dsurefire.timeout=1200
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which TestNG test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the TestNG guide for the patterns behind flaky TestNG suites on any CI: dependency groups, parallel methods, and retry analyzers done wrong.
Frequently asked questions
- Why do my TestNG tests pass locally and fail on GitHub Actions?
- Start with the runner. A private repository's runner has 2 vCPUs, so a parallel suite's 5 threads share 2 CPUs there, and with forkCount=1C, Surefire starts 2 test JVMs instead of one per core of your laptop. Reproduce with mvn test -DforkCount=2, then run the failing class on its own and with the rest of the suite, to see whether it depends on state another class leaves behind.
- Can I keep my IRetryAnalyzer?
- Not on the runs you upload. The attempt it retries is written to Surefire's report as skipped and the retry as a pass, so Test Engine records a skip and a pass and never sees the failure it needs to flag the test. If the analyzer has to stay in the code, make retry() return false when the CI environment variable is set: GitHub Actions sets it to true on every run.
- Why do tests that depend on a flaky test show up as skipped?
- With dependsOnMethods or dependsOnGroups, a method whose dependency failed is not run and is marked SKIP. On an attempt where the flaky test fails, its dependents reach Test Engine as skipped, not failed, and quarantining the flaky test does not change that. If they only need to run after it, alwaysRun = true makes the dependency soft: they run even when it failed.