Flaky RSpec tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, RSpec needs a gem, one require and one secret: add rspec-mergify to the test group, require it from .rspec, and give the step that runs RSpec a MERGIFY_TOKEN. The gem uploads every example's result, applies quarantine inside the run, and reruns new examples before the pull request can merge.
Setup
Run RSpec with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
Gemfile: Add rspec-mergify to the test group of the Gemfile, next to rspec or rspec-rails.group :test do
gem "rspec"
gem "rspec-mergify"
end.rspec: Require rspec_mergify from .rspec, since Bundler does not load it.--require spec_helper
--require rspec_mergify.github/workflows/ci.yml: Pass the MERGIFY_TOKEN secret to the step that runs RSpec.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: ruby/setup-ruby@v1
with:
ruby-version: "3.4"
bundler-cache: true
- name: Run tests
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
run: bundle exec rspecbundle exec rspecdoes not callBundler.require, and the gem's entry file isrspec_mergify. Without the--requireline, RSpec runs as usual and nothing reaches Test Engine.- Lockfile generated on a Mac? Run
bundle lock --add-platform x86_64-linuxand commit it, orbundler-cachefails on the Linux runner before any spec runs. - Running RSpec across a matrix? Every leg reports the same
GITHUB_JOB, so setMERGIFY_TEST_JOB_NAME: test (${{ matrix.ruby-version }})on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken. - CRuby 3.1 to 4.0. A JRuby or
ruby-headleg runs the suite but uploads nothing; give that leg a JUnit report fromrspec_junit_formatterand the upload action instead. - No
continue-on-error: a quarantined example that fails is marked pending, so RSpec's exit code is already the right one.
Full reference: RSpec on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs replays the job from the top: Ruby setup, bundle install from cache, any service containers and database setup, then every example, to retry the one that failed.
rspec-retry retries the example in place and reports the attempt that passed. The flake is absorbed and nothing records it.
rspec-mergify counts those attempts instead of hiding them. On a pull request it reruns the examples the target branch does not have yet, right after its own run, within the Prevention budget, and a failure among rspec-retry's tries counts too: the example is flaky and RSpec reports it failed. On the default branch it reruns the examples Test Engine knows as unhealthy without changing the result.
Runners
Why RSpec tests flake on GitHub Actions
A new random order on every attempt
With
config.order = :random, each attempt of the job runs a new seed, so an order-dependent spec fails on one attempt and passes on the rerun. The seed is printed in the failed attempt's log as "Randomized with seed". Reproduce with it before calling the spec flaky.bundle exec rspec --seed 41853 --bisectCapybara's two-second wait on a two-core runner
Capybara.default_max_wait_timeis 2 seconds. Headless Chrome on a 2-vCPU private runner, sharing it with Rails and Postgres, takes longer to render than your laptop, and system specs time out on whichever page is slowest that day. Raise the wait for CI rather than adding sleeps.# spec/support/capybara.rb Capybara.default_max_wait_time = ENV["CI"] ? 5 : 2Database setup racing the service container
In a Rails app,
bin/rails db:prepareagainst a Postgres service container with no health check can run while the server is still starting. Sometimes it fails outright; sometimes it half-succeeds and the first specs fail on missing tables. A health check makes the job wait for the server before any step runs.services: postgres: image: postgres:17 env: POSTGRES_PASSWORD: postgres ports: ["5432:5432"] options: >- --health-cmd pg_isready --health-interval 5s --health-timeout 5s --health-retries 10
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which RSpec test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the RSpec guide for the patterns behind flaky RSpec suites on any CI: order-dependent specs, database_cleaner strategies, and rspec-retry pitfalls.
Frequently asked questions
- Why do my RSpec specs pass locally and fail on GitHub Actions?
- On GitHub Actions the usual causes are a different random seed on every attempt, Capybara timeouts on a 2-vCPU runner, and a database that was not ready when the suite started. Re-run locally with the seed from the failed attempt's log first.
- Does rspec-mergify replace rspec-retry?
- It does not need to. If you keep rspec-retry, rspec-mergify counts its attempts: a new example that only passed on retry is flaky and fails the pull request. Most teams drop rspec-retry once quarantine handles the known flaky specs.
- Do I need rspec_junit_formatter on GitHub Actions?
- Not with rspec-mergify, which uploads the results itself. Keep the formatter only for legs the gem does not support, such as JRuby, and upload that report with the Mergify upload action.