Flaky Rust tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, Rust tests reach Test Engine as a JUnit report: cargo-nextest writes it from a ci profile, and the mergifyio/gha-mergify-ci action uploads it and decides the job's result with quarantine taken into account. Two lines of that profile matter on CI: the report path, and fail-fast, which nextest turns on by default and which would stop the run before every test has a result to report.
Setup
Run Rust with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
.config/nextest.toml: Add a ci profile that runs every test and writes a JUnit report to target/nextest/ci/junit.xml.[profile.ci]
# Run all tests regardless of failures.
fail-fast = false
[profile.ci.junit]
path = "junit.xml".github/workflows/ci.yml: Install cargo-nextest, run the ci profile, then upload the report with the Mergify action, which sets the job's result.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
env:
HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
steps:
- uses: actions/checkout@v7
- name: Install cargo-nextest
uses: taiki-e/install-action@nextest
- name: Run tests
id: tests
continue-on-error: true
run: cargo nextest run --profile ci
- name: Mergify CI Upload
if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
uses: mergifyio/gha-mergify-ci@v28
with:
token: ${{ secrets.MERGIFY_TOKEN }}
report_path: target/nextest/ci/junit.xml
test_step_outcome: ${{ steps.tests.outcome }}
- name: Fail when tests failed and no report was uploaded
if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
run: exit 1fail-fast = falseis nextest's own advice for a CI profile. With the default, the run stops scheduling tests after the first failure, and a test that never ran has no result for Test Engine.- The JUnit
pathis relative to the profile's directory, so theciprofile writes target/nextest/ci/junit.xml. That is thereport_paththe upload step needs. continue-on-error: trueon the test step is what lets quarantine work: the upload step decides whether the job passes, ignoring failures of quarantined tests.test_step_outcometells it when the test step itself failed, so a crash that left a clean report does not pass for green.- Pull requests from forks and from Dependabot get no
MERGIFY_TOKEN.HAS_MERGIFY_TOKENskips the upload there, and the last step fails the job when tests failed and nothing was uploaded, orcontinue-on-errorwould let those failures through. - The hosted Ubuntu image ships a Rust toolchain but not cargo-nextest. taiki-e/install-action downloads nextest's prebuilt binary instead of compiling it with
cargo install. - Testing several toolchains in a matrix? Every leg reports the same
GITHUB_JOB, so set the upload step'sjob_nameinput per leg, such asjob_name: test (${{ matrix.toolchain }}). Without it, the legs' results merge into one test, and a test that only fails on beta looks flaky instead of broken.
Full reference: Rust on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs starts the job again on a new runner. Unless you cache target/, the crate and its dependencies compile from scratch before every test binary runs again, all to retry one test.
nextest's own retries are cheaper: set retries = 2 in the profile, or pass --retries 2, and a failing test is run again on its own. A test that passes on a retry is marked FLAKY, and by default, if nothing else failed, the run exits 0. The failed attempts are kept in the JUnit report as elements, inside a test case that reads as passed. Since nextest 0.9.131, flaky-result = "fail" fails the run on a flaky test instead.
With Test Engine, every upload adds to each test's history on the default branch and on pull requests, which is how a test that passes and fails on the same commit gets flagged. Once a flaky test is quarantined, the upload step ignores its failure when it sets the job's result, and the test keeps running and reporting until it is healthy again.
Runners
Why Rust tests flake on GitHub Actions
One process per test, two at a time
nextest runs each test in its own process, and test-threads defaults to the number of CPUs: 2 on a private-repository runner, 4 on a public one. Statics and environment variables are no longer shared between tests the way they are under
cargo test, but anything outside the process still is, like a fixed port or the database in a service container. How often two tests collide over it depends on how many run at once, so reproduce with the runner's count.cargo nextest run --profile ci --test-threads 2A hung test that leaves no report
nextest marks a test SLOW after 60 seconds but lets it run unless the profile sets
terminate-after. A test that deadlocks on a two-core runner then runs until the job's timeout-minutes, 360 by default, cancels the job. The upload step only runs on success() or failure(), not on a cancelled job, so that run reports nothing. Terminate the test instead, and it fails under its own name in the report.[profile.ci] fail-fast = false slow-timeout = { period = "60s", terminate-after = 3 }Shards that change with every pull request
--partition slice:1/3 deals tests out round-robin, after filters, so adding a test moves others to a different shard, next to different neighbors. A test that collides with a neighbor over a port or a file can start failing on a pull request that never touched it.
hash:1/3keeps each test in the same bucket whatever is added. Either way, keep the shard index out ofjob_name: a name per shard would split a test's history.cargo nextest run --profile ci --partition hash:1/3
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which Rust test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the Rust guide for the patterns behind flaky Rust suites on any CI: test-thread races, async runtime drift, and cargo test parallelism.
Frequently asked questions
- Why do my Rust tests pass with cargo test and fail with nextest on GitHub Actions?
- cargo test runs a binary's tests as threads of one process, and nextest runs each test in its own process, as many at once as the runner has CPUs (2 on a private repository). State inside the process is shared under cargo test and not under nextest, while a port or a database is shared under both. Reproduce with cargo nextest run --test-threads 2.
- Can I keep cargo test instead of cargo-nextest?
- Yes. cargo2junit converts cargo test's JSON output to JUnit, which needs RUSTC_BOOTSTRAP=1 on a stable toolchain. Add set -o pipefail to the step, or it reports cargo2junit's exit status instead of the test run's, and a failing run can upload as green.
- What name does a Rust test get in Test Engine?
- nextest writes one test suite per test binary and one test case per test, so a test is named after its binary and its path, such as my-crate.tests::parses_header. Renaming the crate, the binary or the module starts a new history for it.
- Why does the upload step need test_step_outcome?
- With continue-on-error on the test step, the upload step decides the job's result. test_step_outcome tells it that cargo nextest failed, so a crash that left a partial but clean junit.xml fails the job instead of passing it.