Skip to content

Flaky Rust tests on GitHub Actions.
Set up, retried right, quarantined.

On GitHub Actions, Rust tests reach Test Engine as a JUnit report: cargo-nextest writes it from a ci profile, and the mergifyio/gha-mergify-ci action uploads it and decides the job's result with quarantine taken into account. Two lines of that profile matter on CI: the report path, and fail-fast, which nextest turns on by default and which would stop the run before every test has a result to report.

By Julien Danjou, Co-founder & CEO of MergifyUpdated

Setup

Run Rust with Test Engine on GitHub Actions

Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:

.config/nextest.toml: Add a ci profile that runs every test and writes a JUnit report to target/nextest/ci/junit.xml.
[profile.ci]
# Run all tests regardless of failures.
fail-fast = false

[profile.ci.junit]
path = "junit.xml"
.github/workflows/ci.yml: Install cargo-nextest, run the ci profile, then upload the report with the Mergify action, which sets the job's result.
name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
    steps:
      - uses: actions/checkout@v7
      - name: Install cargo-nextest
        uses: taiki-e/install-action@nextest
      - name: Run tests
        id: tests
        continue-on-error: true
        run: cargo nextest run --profile ci
      - name: Mergify CI Upload
        if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
        uses: mergifyio/gha-mergify-ci@v28
        with:
          token: ${{ secrets.MERGIFY_TOKEN }}
          report_path: target/nextest/ci/junit.xml
          test_step_outcome: ${{ steps.tests.outcome }}
      - name: Fail when tests failed and no report was uploaded
        if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
        run: exit 1
  • fail-fast = false is nextest's own advice for a CI profile. With the default, the run stops scheduling tests after the first failure, and a test that never ran has no result for Test Engine.
  • The JUnit path is relative to the profile's directory, so the ci profile writes target/nextest/ci/junit.xml. That is the report_path the upload step needs.
  • continue-on-error: true on the test step is what lets quarantine work: the upload step decides whether the job passes, ignoring failures of quarantined tests. test_step_outcome tells it when the test step itself failed, so a crash that left a clean report does not pass for green.
  • Pull requests from forks and from Dependabot get no MERGIFY_TOKEN. HAS_MERGIFY_TOKEN skips the upload there, and the last step fails the job when tests failed and nothing was uploaded, or continue-on-error would let those failures through.
  • The hosted Ubuntu image ships a Rust toolchain but not cargo-nextest. taiki-e/install-action downloads nextest's prebuilt binary instead of compiling it with cargo install.
  • Testing several toolchains in a matrix? Every leg reports the same GITHUB_JOB, so set the upload step's job_name input per leg, such as job_name: test (${{ matrix.toolchain }}). Without it, the legs' results merge into one test, and a test that only fails on beta looks flaky instead of broken.

Full reference: Rust on docs.mergify.com.

Retries

Re-run failed jobs versus retrying the test

Re-run failed jobs starts the job again on a new runner. Unless you cache target/, the crate and its dependencies compile from scratch before every test binary runs again, all to retry one test.

nextest's own retries are cheaper: set retries = 2 in the profile, or pass --retries 2, and a failing test is run again on its own. A test that passes on a retry is marked FLAKY, and by default, if nothing else failed, the run exits 0. The failed attempts are kept in the JUnit report as elements, inside a test case that reads as passed. Since nextest 0.9.131, flaky-result = "fail" fails the run on a flaky test instead.

With Test Engine, every upload adds to each test's history on the default branch and on pull requests, which is how a test that passes and fails on the same commit gets flagged. Once a flaky test is quarantined, the upload step ignores its failure when it sets the job's result, and the test keeps running and reporting until it is healthy again.

Runners

Why Rust tests flake on GitHub Actions

  1. One process per test, two at a time

    nextest runs each test in its own process, and test-threads defaults to the number of CPUs: 2 on a private-repository runner, 4 on a public one. Statics and environment variables are no longer shared between tests the way they are under cargo test, but anything outside the process still is, like a fixed port or the database in a service container. How often two tests collide over it depends on how many run at once, so reproduce with the runner's count.

    cargo nextest run --profile ci --test-threads 2
  2. A hung test that leaves no report

    nextest marks a test SLOW after 60 seconds but lets it run unless the profile sets terminate-after. A test that deadlocks on a two-core runner then runs until the job's timeout-minutes, 360 by default, cancels the job. The upload step only runs on success() or failure(), not on a cancelled job, so that run reports nothing. Terminate the test instead, and it fails under its own name in the report.

    [profile.ci]
    fail-fast = false
    slow-timeout = { period = "60s", terminate-after = 3 }
  3. Shards that change with every pull request

    --partition slice:1/3 deals tests out round-robin, after filters, so adding a test moves others to a different shard, next to different neighbors. A test that collides with a neighbor over a port or a file can start failing on a pull request that never touched it. hash:1/3 keeps each test in the same bucket whatever is added. Either way, keep the shard index out of job_name: a name per shard would split a test's history.

    cargo nextest run --profile ci --partition hash:1/3

The missing layer

GitHub Actions doesn't know what a test is.

GitHub Actions tells you a job failed. Test Engine tells you which Rust test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the Rust guide for the patterns behind flaky Rust suites on any CI: test-thread races, async runtime drift, and cargo test parallelism.

Frequently asked questions

Why do my Rust tests pass with cargo test and fail with nextest on GitHub Actions?
cargo test runs a binary's tests as threads of one process, and nextest runs each test in its own process, as many at once as the runner has CPUs (2 on a private repository). State inside the process is shared under cargo test and not under nextest, while a port or a database is shared under both. Reproduce with cargo nextest run --test-threads 2.
Can I keep cargo test instead of cargo-nextest?
Yes. cargo2junit converts cargo test's JSON output to JUnit, which needs RUSTC_BOOTSTRAP=1 on a stable toolchain. Add set -o pipefail to the step, or it reports cargo2junit's exit status instead of the test run's, and a failing run can upload as green.
What name does a Rust test get in Test Engine?
nextest writes one test suite per test binary and one test case per test, so a test is named after its binary and its path, such as my-crate.tests::parses_header. Renaming the crate, the binary or the module starts a new history for it.
Why does the upload step need test_step_outcome?
With continue-on-error on the test step, the upload step decides the job's result. test_step_outcome tells it that cargo nextest failed, so a crash that left a partial but clean junit.xml fails the job instead of passing it.

Catch the flaky Rust tests in your GitHub Actions runs.

Mergify merges more than half a million pull requests a month without breaking main.