Flaky NUnit tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, NUnit results reach Test Engine as a JUnit report. Add JunitXml.TestLogger to the test project so dotnet test writes junit.xml, then upload it with the mergifyio/gha-mergify-ci action, which also decides whether the job passes once quarantined tests are set aside. The detail to get right is where the report lands: the logger resolves its path from the test project's folder, not from the repository root.
Setup
Run NUnit with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
Demo.Tests.csproj: Add JunitXml.TestLogger to the test project, next to NUnit and its test adapter.<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.11.0" />
<PackageReference Include="NUnit" Version="5.0.0" />
<PackageReference Include="NUnit3TestAdapter" Version="6.3.1" />
<PackageReference Include="JunitXml.TestLogger" Version="8.1.0" />
</ItemGroup>
</Project>.github/workflows/ci.yml: Write junit.xml with the JUnit logger, then upload it with the Mergify action, which sets the job's result.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
env:
HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
steps:
- uses: actions/checkout@v7
- uses: actions/setup-dotnet@v6
with:
dotnet-version: 10.0.x
- name: Run tests
id: tests
continue-on-error: true
run: dotnet test --logger "junit;LogFilePath=junit.xml"
- name: Mergify CI Upload
if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
uses: mergifyio/gha-mergify-ci@v28
with:
token: ${{ secrets.MERGIFY_TOKEN }}
report_path: junit.xml
test_step_outcome: ${{ steps.tests.outcome }}
- name: Fail when tests failed and no report was uploaded
if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
run: exit 1LogFilePathis resolved from the folder that holds the test project's.csproj, not from the repository root. The project above sits at the root, soreport_path: junit.xmlmatches. For a project intests/Api.Tests/, uploadtests/Api.Tests/junit.xml.- A solution with several test projects writes one report per project, all with the same name. Give the logger an absolute path with the {assembly} token,
--logger "junit;LogFilePath=${{ github.workspace }}/test-results/{assembly}.xml", and uploadtest-results/*.xml. continue-on-error: truehands the job's result to the upload step, which leaves out failures of quarantined tests.dotnet testexits 1 for a failing test, and also when a test assembly cannot load or the run aborts.test_step_outcomemakes sure that second kind fails the job even when the report it left behind looks clean.- Pull requests from forks and from Dependabot run without
MERGIFY_TOKEN.HAS_MERGIFY_TOKENskips the upload there, and the last step fails the job when tests failed and nothing was uploaded. - Turned on
EnableNUnitRunner? On its own it changes nothing fordotnet test. AddTestingPlatformDotnetTestSupportand the tests run on Microsoft.Testing.Platform, which silently ignores VSTest options such as--logger: the suite runs, and nojunit.xmlappears.
Full reference: NUnit on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs starts again from the NuGet restore and the build, then runs every fixture to retry the one test that failed. The pull request keeps whichever attempt comes back green.
NUnit's [Retry(3)] retries inside the run instead. The number is the total count of attempts, and only an assertion failure triggers another one: a test that throws an unexpected exception is reported as an error and not retried, unless that exception type is listed in RetryExceptions (NUnit 4.5 and later). When a later attempt passes, the test counts as passed and the race stays in the code.
Test Engine records every run instead of keeping the lucky one. Each upload adds to each test's history, on pull requests and on the default branch, so a test that fails and passes on the same commit is flagged as flaky. Once it is quarantined, the upload step leaves its failure out of the job's result, and the test keeps running and reporting until it is healthy again.
Runners
Why NUnit tests flake on GitHub Actions
Two workers on a private-repository runner
Unless the assembly sets
[LevelOfParallelism], NUnit runs[Parallelizable]tests on as many workers as the machine has processors, or 2, whichever is greater. That is 2 on a private-repository runner and 4 on a public one, whatever your laptop has. Fixtures that share a static field or a database row interleave differently at each width, so a race that never shows on your machine shows on CI, and the other way around. Reproduce with the runner's worker count, which the NUnit adapter reads from the run settings.dotnet test -- NUnit.NumberOfTestWorkers=2A hung test that holds the job for six hours
A test that deadlocks, typically on
.Resultfor a task that needs the thread it is blocking, keeps the job running until GitHub cancels it, after 360 minutes by default. Once the job is cancelled, neither success() nor failure() is true, so the upload step never runs and the hang leaves no result in Test Engine.--blame-hang-timeouthas VSTest dump and stop the test host when one test runs past the limit, so the test step fails in minutes and the dump shows where it was stuck.dotnet test --logger "junit;LogFilePath=junit.xml" --blame-hang-timeout 10mTwo target frameworks running at once
Since the .NET 9 SDK, a project with
<TargetFrameworks>net8.0;net10.0</TargetFrameworks>runs its tests for both frameworks in parallel by default. On a 2-vCPU runner, the two runs compete for the CPUs and for whatever the suite keeps outside the process, such as the same database or the same port. A test that only fails while the other framework's run touches the same row looks flaky under both. Give each framework its own resources, or run them one after the other. The report needs the{framework}token in its file name too, or one framework's report overwrites the other's.dotnet test -p:TestTfmsInParallel=false
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which NUnit test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the NUnit guide for the patterns behind flaky NUnit suites on any CI: parallelizableAttribute, TestContext races, and fixture reuse pitfalls.
Frequently asked questions
- Why do my NUnit tests pass locally and fail on GitHub Actions?
- Start with the runner. NUnit sizes its worker pool from the CPU count, so a private repository's 2-vCPU runner runs your parallel fixtures two at a time, and a multi-targeted project runs its frameworks side by side on those same 2 vCPUs. Reproduce with dotnet test -- NUnit.NumberOfTestWorkers=2 and -p:TestTfmsInParallel=false before you call a test flaky.
- Can I keep the NUnit Console Runner instead of dotnet test?
- Yes, with one more file. The console runner only writes nunit3 or nunit2 XML, so pass --result=junit.xml;transform=nunit3-junit.xslt with the XSLT NUnit publishes in its nunit-transforms repository, and quote the whole option, because the semicolon is a command separator in most shells. dotnet test with the JUnit logger needs no transform.
- Why is there no junit.xml after dotnet test?
- Look in the test project's folder first: LogFilePath is relative to the .csproj, not to the repository root. If the file is nowhere, check whether the project sets TestingPlatformDotnetTestSupport. With it, dotnet test runs the tests on Microsoft.Testing.Platform and ignores --logger without an error.