Skip to content

Flaky RSpec tests on GitHub Actions.
Set up, retried right, quarantined.

On GitHub Actions, RSpec needs a gem, one require and one secret: add rspec-mergify to the test group, require it from .rspec, and give the step that runs RSpec a MERGIFY_TOKEN. The gem uploads every example's result, applies quarantine inside the run, and reruns new examples before the pull request can merge.

By Julien Danjou, Co-founder & CEO of MergifyUpdated

Setup

Run RSpec with Test Engine on GitHub Actions

Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:

Gemfile: Add rspec-mergify to the test group of the Gemfile, next to rspec or rspec-rails.
group :test do
  gem "rspec"
  gem "rspec-mergify"
end
.rspec: Require rspec_mergify from .rspec, since Bundler does not load it.
--require spec_helper
--require rspec_mergify
.github/workflows/ci.yml: Pass the MERGIFY_TOKEN secret to the step that runs RSpec.
name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: "3.4"
          bundler-cache: true
      - name: Run tests
        env:
          MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
        run: bundle exec rspec
  • bundle exec rspec does not call Bundler.require, and the gem's entry file is rspec_mergify. Without the --require line, RSpec runs as usual and nothing reaches Test Engine.
  • Lockfile generated on a Mac? Run bundle lock --add-platform x86_64-linux and commit it, or bundler-cache fails on the Linux runner before any spec runs.
  • Running RSpec across a matrix? Every leg reports the same GITHUB_JOB, so set MERGIFY_TEST_JOB_NAME: test (${{ matrix.ruby-version }}) on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken.
  • CRuby 3.1 to 4.0. A JRuby or ruby-head leg runs the suite but uploads nothing; give that leg a JUnit report from rspec_junit_formatter and the upload action instead.
  • No continue-on-error: a quarantined example that fails is marked pending, so RSpec's exit code is already the right one.

Full reference: RSpec on docs.mergify.com.

Retries

Re-run failed jobs versus retrying the test

Re-run failed jobs replays the job from the top: Ruby setup, bundle install from cache, any service containers and database setup, then every example, to retry the one that failed.

rspec-retry retries the example in place and reports the attempt that passed. The flake is absorbed and nothing records it.

rspec-mergify counts those attempts instead of hiding them. On a pull request it reruns the examples the target branch does not have yet, right after its own run, within the Prevention budget, and a failure among rspec-retry's tries counts too: the example is flaky and RSpec reports it failed. On the default branch it reruns the examples Test Engine knows as unhealthy without changing the result.

Runners

Why RSpec tests flake on GitHub Actions

  1. A new random order on every attempt

    With config.order = :random, each attempt of the job runs a new seed, so an order-dependent spec fails on one attempt and passes on the rerun. The seed is printed in the failed attempt's log as "Randomized with seed". Reproduce with it before calling the spec flaky.

    bundle exec rspec --seed 41853 --bisect
  2. Capybara's two-second wait on a two-core runner

    Capybara.default_max_wait_time is 2 seconds. Headless Chrome on a 2-vCPU private runner, sharing it with Rails and Postgres, takes longer to render than your laptop, and system specs time out on whichever page is slowest that day. Raise the wait for CI rather than adding sleeps.

    # spec/support/capybara.rb
    Capybara.default_max_wait_time = ENV["CI"] ? 5 : 2
  3. Database setup racing the service container

    In a Rails app, bin/rails db:prepare against a Postgres service container with no health check can run while the server is still starting. Sometimes it fails outright; sometimes it half-succeeds and the first specs fail on missing tables. A health check makes the job wait for the server before any step runs.

    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_PASSWORD: postgres
        ports: ["5432:5432"]
        options: >-
          --health-cmd pg_isready
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10

The missing layer

GitHub Actions doesn't know what a test is.

GitHub Actions tells you a job failed. Test Engine tells you which RSpec test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the RSpec guide for the patterns behind flaky RSpec suites on any CI: order-dependent specs, database_cleaner strategies, and rspec-retry pitfalls.

Frequently asked questions

Why do my RSpec specs pass locally and fail on GitHub Actions?
On GitHub Actions the usual causes are a different random seed on every attempt, Capybara timeouts on a 2-vCPU runner, and a database that was not ready when the suite started. Re-run locally with the seed from the failed attempt's log first.
Does rspec-mergify replace rspec-retry?
It does not need to. If you keep rspec-retry, rspec-mergify counts its attempts: a new example that only passed on retry is flaky and fails the pull request. Most teams drop rspec-retry once quarantine handles the known flaky specs.
Do I need rspec_junit_formatter on GitHub Actions?
Not with rspec-mergify, which uploads the results itself. Keep the formatter only for legs the gem does not support, such as JRuby, and upload that report with the Mergify upload action.

Catch the flaky RSpec tests in your GitHub Actions runs.

Mergify merges more than half a million pull requests a month without breaking main.