Skip to content

Flaky NUnit tests on GitHub Actions.
Set up, retried right, quarantined.

On GitHub Actions, NUnit results reach Test Engine as a JUnit report. Add JunitXml.TestLogger to the test project so dotnet test writes junit.xml, then upload it with the mergifyio/gha-mergify-ci action, which also decides whether the job passes once quarantined tests are set aside. The detail to get right is where the report lands: the logger resolves its path from the test project's folder, not from the repository root.

By Julien Danjou, Co-founder & CEO of MergifyUpdated

Setup

Run NUnit with Test Engine on GitHub Actions

Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:

Demo.Tests.csproj: Add JunitXml.TestLogger to the test project, next to NUnit and its test adapter.
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.11.0" />
    <PackageReference Include="NUnit" Version="5.0.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="6.3.1" />
    <PackageReference Include="JunitXml.TestLogger" Version="8.1.0" />
  </ItemGroup>

</Project>
.github/workflows/ci.yml: Write junit.xml with the JUnit logger, then upload it with the Mergify action, which sets the job's result.
name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-dotnet@v6
        with:
          dotnet-version: 10.0.x
      - name: Run tests
        id: tests
        continue-on-error: true
        run: dotnet test --logger "junit;LogFilePath=junit.xml"
      - name: Mergify CI Upload
        if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
        uses: mergifyio/gha-mergify-ci@v28
        with:
          token: ${{ secrets.MERGIFY_TOKEN }}
          report_path: junit.xml
          test_step_outcome: ${{ steps.tests.outcome }}
      - name: Fail when tests failed and no report was uploaded
        if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
        run: exit 1
  • LogFilePath is resolved from the folder that holds the test project's .csproj, not from the repository root. The project above sits at the root, so report_path: junit.xml matches. For a project in tests/Api.Tests/, upload tests/Api.Tests/junit.xml.
  • A solution with several test projects writes one report per project, all with the same name. Give the logger an absolute path with the {assembly} token, --logger "junit;LogFilePath=${{ github.workspace }}/test-results/{assembly}.xml", and upload test-results/*.xml.
  • continue-on-error: true hands the job's result to the upload step, which leaves out failures of quarantined tests. dotnet test exits 1 for a failing test, and also when a test assembly cannot load or the run aborts. test_step_outcome makes sure that second kind fails the job even when the report it left behind looks clean.
  • Pull requests from forks and from Dependabot run without MERGIFY_TOKEN. HAS_MERGIFY_TOKEN skips the upload there, and the last step fails the job when tests failed and nothing was uploaded.
  • Turned on EnableNUnitRunner? On its own it changes nothing for dotnet test. Add TestingPlatformDotnetTestSupport and the tests run on Microsoft.Testing.Platform, which silently ignores VSTest options such as --logger: the suite runs, and no junit.xml appears.

Full reference: NUnit on docs.mergify.com.

Retries

Re-run failed jobs versus retrying the test

Re-run failed jobs starts again from the NuGet restore and the build, then runs every fixture to retry the one test that failed. The pull request keeps whichever attempt comes back green.

NUnit's [Retry(3)] retries inside the run instead. The number is the total count of attempts, and only an assertion failure triggers another one: a test that throws an unexpected exception is reported as an error and not retried, unless that exception type is listed in RetryExceptions (NUnit 4.5 and later). When a later attempt passes, the test counts as passed and the race stays in the code.

Test Engine records every run instead of keeping the lucky one. Each upload adds to each test's history, on pull requests and on the default branch, so a test that fails and passes on the same commit is flagged as flaky. Once it is quarantined, the upload step leaves its failure out of the job's result, and the test keeps running and reporting until it is healthy again.

Runners

Why NUnit tests flake on GitHub Actions

  1. Two workers on a private-repository runner

    Unless the assembly sets [LevelOfParallelism], NUnit runs [Parallelizable] tests on as many workers as the machine has processors, or 2, whichever is greater. That is 2 on a private-repository runner and 4 on a public one, whatever your laptop has. Fixtures that share a static field or a database row interleave differently at each width, so a race that never shows on your machine shows on CI, and the other way around. Reproduce with the runner's worker count, which the NUnit adapter reads from the run settings.

    dotnet test -- NUnit.NumberOfTestWorkers=2
  2. A hung test that holds the job for six hours

    A test that deadlocks, typically on .Result for a task that needs the thread it is blocking, keeps the job running until GitHub cancels it, after 360 minutes by default. Once the job is cancelled, neither success() nor failure() is true, so the upload step never runs and the hang leaves no result in Test Engine. --blame-hang-timeout has VSTest dump and stop the test host when one test runs past the limit, so the test step fails in minutes and the dump shows where it was stuck.

    dotnet test --logger "junit;LogFilePath=junit.xml" --blame-hang-timeout 10m
  3. Two target frameworks running at once

    Since the .NET 9 SDK, a project with <TargetFrameworks>net8.0;net10.0</TargetFrameworks> runs its tests for both frameworks in parallel by default. On a 2-vCPU runner, the two runs compete for the CPUs and for whatever the suite keeps outside the process, such as the same database or the same port. A test that only fails while the other framework's run touches the same row looks flaky under both. Give each framework its own resources, or run them one after the other. The report needs the {framework} token in its file name too, or one framework's report overwrites the other's.

    dotnet test -p:TestTfmsInParallel=false

The missing layer

GitHub Actions doesn't know what a test is.

GitHub Actions tells you a job failed. Test Engine tells you which NUnit test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the NUnit guide for the patterns behind flaky NUnit suites on any CI: parallelizableAttribute, TestContext races, and fixture reuse pitfalls.

Frequently asked questions

Why do my NUnit tests pass locally and fail on GitHub Actions?
Start with the runner. NUnit sizes its worker pool from the CPU count, so a private repository's 2-vCPU runner runs your parallel fixtures two at a time, and a multi-targeted project runs its frameworks side by side on those same 2 vCPUs. Reproduce with dotnet test -- NUnit.NumberOfTestWorkers=2 and -p:TestTfmsInParallel=false before you call a test flaky.
Can I keep the NUnit Console Runner instead of dotnet test?
Yes, with one more file. The console runner only writes nunit3 or nunit2 XML, so pass --result=junit.xml;transform=nunit3-junit.xslt with the XSLT NUnit publishes in its nunit-transforms repository, and quote the whole option, because the semicolon is a command separator in most shells. dotnet test with the JUnit logger needs no transform.
Why is there no junit.xml after dotnet test?
Look in the test project's folder first: LogFilePath is relative to the .csproj, not to the repository root. If the file is nowhere, check whether the project sets TestingPlatformDotnetTestSupport. With it, dotnet test runs the tests on Microsoft.Testing.Platform and ignores --logger without an error.

Catch the flaky NUnit tests in your GitHub Actions runs.

Mergify merges more than half a million pull requests a month without breaking main.