Skip to content

Flaky MSTest tests on GitHub Actions.
Set up, retried right, quarantined.

MSTest reaches Test Engine on GitHub Actions through a JUnit report. JunitXml.TestLogger writes junit.xml during dotnet test, and the mergifyio/gha-mergify-ci action uploads it, then sets the job's result with the failures of quarantined tests left out. Before anything else, check which test platform the project runs on: on MSTest.Sdk, dotnet test runs Microsoft.Testing.Platform, which drops --logger without an error and leaves the upload step nothing to send.

By Julien Danjou, Co-founder & CEO of MergifyUpdated

Setup

Run MSTest with Test Engine on GitHub Actions

Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:

Demo.Tests.csproj: Reference JunitXml.TestLogger next to the MSTest package, on Microsoft.NET.Sdk, where dotnet test runs on VSTest.
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="MSTest" Version="4.5.2" />
    <PackageReference Include="JunitXml.TestLogger" Version="8.1.0" />
  </ItemGroup>

</Project>
.github/workflows/ci.yml: Run dotnet test with the JUnit logger, then let the Mergify action upload junit.xml and decide the job's result.
name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-dotnet@v6
        with:
          dotnet-version: 10.0.x
      - name: Run tests
        id: tests
        continue-on-error: true
        run: dotnet test --logger "junit;LogFilePath=junit.xml"
      - name: Mergify CI Upload
        if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
        uses: mergifyio/gha-mergify-ci@v28
        with:
          token: ${{ secrets.MERGIFY_TOKEN }}
          report_path: junit.xml
          test_step_outcome: ${{ steps.tests.outcome }}
      - name: Fail when tests failed and no report was uploaded
        if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
        run: exit 1
  • MSTest.Sdk sets EnableMSTestRunner and TestingPlatformDotnetTestSupport to true, so its tests run on Microsoft.Testing.Platform, under dotnet test too. In that mode, VSTest options such as --logger are silently ignored: the tests pass or fail as usual and junit.xml is never written. Add <UseVSTest>true</UseVSTest> to the project to switch it back. The project above uses the MSTest package on Microsoft.NET.Sdk, which stays on VSTest as long as nothing sets TestingPlatformDotnetTestSupport.
  • The logger resolves LogFilePath from the test project's folder. With the .csproj at the repository root, as here, report_path: junit.xml finds the file. For a project in tests/Shop.Tests/, upload tests/Shop.Tests/junit.xml.
  • The upload step owns the job's result. continue-on-error: true stops a red test step from failing the job by itself, and the upload step fails it unless every failing test is quarantined. test_step_outcome passes it the test step's real outcome, so a test host that crashed after writing part of the report still fails the job, even when that report shows no failure.
  • MERGIFY_TOKEN is empty on pull requests from forks and from Dependabot. HAS_MERGIFY_TOKEN skips the upload on those runs, and the last step fails the job when the tests failed, which continue-on-error would otherwise let through.
  • A --filter that matches no test still exits 0 with a warning, and the upload step receives a report with no test in it. Set RunConfiguration.TreatNoTestsAsError to true in a .runsettings file passed with --settings, so a typo in a filter fails the job instead of passing it with nothing tested.
  • Running the suite on ubuntu-latest and windows-latest from one matrix? Both legs report the same GITHUB_JOB, so add job_name: test (${{ matrix.os }}) to the upload step. Without it the two legs share one history per test, and a test that only fails on Windows looks flaky instead of broken on Windows.

Full reference: MSTest on docs.mergify.com.

Retries

Re-run failed jobs versus retrying the test

Re-run failed jobs starts the test job again on the same commit: setup-dotnet, the NuGet restore, the build, then every test in the assembly, to find out whether one method was unlucky. The pull request shows the attempt that comes back green. The red one stays in the run's history, which nobody opens once the check has passed.

MSTest has had its own [Retry] since 3.8, on a test method or, from MSTest 4.3, on a whole class. [Retry(3)] runs a test that fails or times out up to three more times, with an optional delay and exponential backoff between attempts. On VSTest, where the JUnit logger runs, only the final result is reported: a test that failed once and then passed reaches junit.xml as a plain pass, and the failure is gone from every report.

Test Engine works from the failures [Retry] would have hidden. Every upload adds each test's result to its history, on pull requests and on the default branch, and a test that fails and passes on the same commit is flagged as flaky. Take [Retry] off the tests you want watched and quarantine the ones you already know are flaky: the upload step leaves a quarantined test's failure out of the job's result, and the test keeps running and reporting until it is healthy again.

Runners

Why MSTest tests flake on GitHub Actions

  1. Method-level parallelism at the runner's width

    [assembly: Parallelize] without a Workers value runs one worker per logical processor. A private-repository runner gives it 2 workers and a public one 4, usually fewer than your laptop. With Scope = ExecutionScope.MethodLevel, any two test methods can overlap, in the same class or not, and which ones do depends on the worker count. Two tests that write the same database row may never overlap on your machine and collide on CI, or the reverse. Pin the width to reproduce the runner. On MSTest 4.4 and later, give the tests that share one resource the same [ResourceLock] key: MSTest then runs those tests one at a time and keeps the rest of the suite parallel.

    // Reproduce a private-repository runner's width locally
    [assembly: Parallelize(Workers = 2, Scope = ExecutionScope.MethodLevel)]
  2. A [Timeout] that fires on CI and keeps running

    [Timeout(2000)] is in milliseconds, and a budget that fits a laptop can be too short on a 2-vCPU runner. What happens after the timeout does more damage than the failure itself. By default, MSTest runs a timed test in a separate task, and when time is up it stops watching that task without stopping it. The test goes on writing to whatever it shares with the tests that run after it, and the next failure lands on a test that did nothing wrong. Since MSTest 3.6, CooperativeCancellation = true cancels a token instead. Pass that token all the way down, so a timed-out test actually stops.

    [TestMethod]
    [Timeout(10_000, CooperativeCancellation = true)]
    public async Task ImportsOrders(CancellationToken cancellationToken)
    {
        await importer.RunAsync(cancellationToken);
    }
  3. A shuffled order that changes on every attempt

    MSTest 4.3 added RandomizeTestOrder, which runs tests in a random order to bring out tests that depend on what ran before them. Without RandomTestOrderSeed, each run draws a new order, so the attempt you start with Re-run failed jobs runs a different sequence and can pass for reasons nobody sees. Seed it with github.run_number, which GitHub keeps on a re-run and increments for each new run. Every attempt of a run then replays one order, and the run number is all you need to replay that order locally.

    - name: Run tests
      id: tests
      continue-on-error: true
      run: dotnet test --logger "junit;LogFilePath=junit.xml" -- MSTest.RandomizeTestOrder=true MSTest.RandomTestOrderSeed=${{ github.run_number }}

The missing layer

GitHub Actions doesn't know what a test is.

GitHub Actions tells you a job failed. Test Engine tells you which MSTest test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the MSTest guide for the patterns behind flaky MSTest suites on any CI: assembly initialization, DeploymentItem races, and parallel test settings.

Frequently asked questions

Why do my MSTest tests pass locally and fail on GitHub Actions?
Start with the worker count. [assembly: Parallelize] uses one worker per logical processor, which is 2 on a private repository's runner, so method-level tests overlap in different pairs than on your laptop. Then look at [Timeout] values in milliseconds that were tuned on a faster machine. Pin Workers = 2 and run the failing class locally before you call a test flaky.
Why does my MSTest.Sdk project write no junit.xml?
MSTest.Sdk runs dotnet test on Microsoft.Testing.Platform, which drops VSTest options like --logger without failing the run. From MTP 1.9 the log carries an MTP0001 warning for it. Set UseVSTest to true in the project file and the JUnit logger runs again. If the file is still missing, look in the test project's folder: LogFilePath is relative to the .csproj, not to the repository root.
Does MSTest's [Retry] attribute hide flaky tests?
On VSTest it does. Only the last attempt of a retried test is reported, so a test that failed and then passed reaches junit.xml as passed, and Test Engine never sees the failure. Keep [Retry] off the tests you want watched and quarantine the known flaky ones instead: they keep running and reporting, and their failures stop failing the job.
Can I skip a flaky test on CI with [CICondition]?
You can: [CICondition(ConditionMode.Exclude)], in MSTest 3.10 and later, skips the test whenever GITHUB_ACTIONS or CI is true, and GitHub Actions sets both on every run. The test then never runs on CI, so nothing shows when it is fixed and nothing catches a regression in the code it covers. Quarantine keeps it running and reporting while its failures stop counting.

Catch the flaky MSTest tests in your GitHub Actions runs.

Mergify merges more than half a million pull requests a month without breaking main.