Flaky Minitest tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, Minitest needs a gem, one require and one secret: add minitest-mergify to the test group, require minitest/mergify from your test helper, and give the step that runs the tests a MERGIFY_TOKEN. The plugin uploads every test's result when the run ends, applies quarantine inside Minitest, and reruns new tests before the pull request can merge. Rails test suites run on Minitest, so the same setup covers bin/rails test.
Setup
Run Minitest with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
Gemfile: Add minitest-mergify to the test group of the Gemfile. In a Rails app, minitest is already there.source 'https://rubygems.org'
group :test do
gem 'minitest'
gem 'minitest-mergify'
gem 'rake'
endtest/test_helper.rb: Require the plugin in the test helper, before any test is defined. Minitest 6 does not load it otherwise.require 'minitest/autorun'
require 'minitest/mergify'
require 'calc'.github/workflows/ci.yml: Pass the MERGIFY_TOKEN secret to the step that runs Minitest. In a Rails app, keep bin/rails test.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: ruby/setup-ruby@v1
with:
ruby-version: "3.4"
bundler-cache: true
- name: Run tests
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
run: bundle exec rake test- Minitest 6 loads only the plugins a suite asks for, so without
require 'minitest/mergify'the run goes as usual and nothing reaches Test Engine. Keep the line on Minitest 5 too: there the plugin would load only after the tests are defined, and spec tests with no body lose their line number. - No
continue-on-errorand notest_step_outcome: there is no upload step to hand the result to. A quarantined test that fails counts as a skip, so the step's exit code is already the job's result. A test file that fails to load still fails the step. - Running Minitest across a matrix? Every leg reports the same
GITHUB_JOB, so setMERGIFY_TEST_JOB_NAME: test (${{ matrix.ruby-version }})on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken. - CRuby 3.1 to 4.0, with Minitest 5.19 or later, or Minitest 6 on Ruby 3.2 and later. A JRuby or
ruby-headleg runs its tests but reports nothing; give that leg the JUnit reporter fromminitest-reportersand themergifyio/gha-mergify-ciupload step instead. - The gem runs on
mergify-ci-core, precompiled per platform. If yourGemfile.lockwas written on a Mac, runbundle lock --add-platform x86_64-linuxand commit it, so Bundler can install the Linux build on the runner. - bin/rails test leaves system tests out. Run
bin/rails test:allin the step if you want them reported too.
Full reference: Minitest on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs replays the job from the top: setup-ruby, the bundle restored from cache, the database setup if you have one, then the whole suite. Minitest also picks a new seed for that attempt (the log prints it as Run options: --seed), so the attempt that passes did not even run the tests in the same order as the one that failed.
minitest-retry moves the retry inside the run: a failing test is retried up to 3 times by default, and it counts as passed if one attempt passes. The [MinitestRetry] retry lines in that job's log are the only trace of the flake.
minitest-mergify reruns with a narrower rule. On a pull request, once the suite is over, it reruns the tests the target branch does not have yet, within the Prevention budget, through Minitest itself, so a test that ran in Rails' parallel workers is rerun there. If one attempt fails, the test is flaky and the job fails. On pushes to the default branch it reruns the tests Test Engine knows as unhealthy, within the Detection budget, without changing the result.
Runners
Why Minitest tests flake on GitHub Actions
A worker count that follows the runner
parallelize(workers: :number_of_processors)forks one worker per core, which is 2 on a private-repository runner and 4 on a public one, against often 10 or more on a laptop. Rails also skips parallelization below 50 tests, so the test file you run locally usually runs in one process, while CI runs the full suite across workers, each with its own share of the tests. Reproduce with the runner's worker count and the seed from the failed attempt's log.PARALLEL_WORKERS=2 bin/rails test --seed 39840Workers that share everything but the database
Active Record gives each worker its own database, suffixed with the worker's number (
test-database-0,test-database-1). The rest is yours to split. Active Storage's Disk service folder or a Redis service container is shared by every worker of the job, so one worker's teardown can delete the files another worker's test is still reading. The Active Storage guide gives each worker its own folder inparallelize_setup.class ActionDispatch::IntegrationTest parallelize_setup do |i| ActiveStorage::Blob.service.root = "#{ActiveStorage::Blob.service.root}-#{i}" end endA new seed on every attempt
Test methods run in random order, and Minitest prints the seed before the first one. Every attempt of the job draws a new seed, so a test that depends on what ran before it fails on one attempt and passes on the next, and Re-run failed jobs reads that as a fluke. Rerun with the failed attempt's seed (
--seed, orSEED=with Rake) before calling the test flaky.SEED=39840 bundle exec rake test
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which Minitest test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the Minitest guide for the patterns behind flaky Minitest suites on any CI: parallelization races, fixture loading, and test isolation in Rails suites.
Frequently asked questions
- Why do my Rails tests pass locally and fail on GitHub Actions?
- Usually the runner, not the test: 2 parallel workers on a private repository, a new seed on every attempt, and a full suite that runs in parallel while the one file you ran locally did not (Rails only parallelizes 50 tests or more). Run the suite with PARALLEL_WORKERS=2 and the seed printed in the failed attempt's log.
- Why does Minitest report 0 failures when the job failed?
- On a pull request, minitest-mergify reruns new tests after the suite. Minitest's summary and your other reporters only show your own run, so a new test that failed on a rerun still counts as passed there. The exit code and the plugin's # Mergify CI: section in the log say it is flaky.
- Do I still need minitest-reporters or minitest-junit on GitHub Actions?
- Not for Test Engine: minitest-mergify uploads the results itself. Keep the JUnit reporter only for a leg the plugin cannot report from, such as JRuby. Moving from a JUnit upload keeps your quarantine entries, while the tests' history starts again under their new names.
- Does minitest-mergify work with Rails parallel tests?
- Yes, whether the tests run in Minitest's threads (parallelize_me!) or in Rails' parallel workers, and its reruns go through those workers too. Quarantine applies the same way as in a single process.