Skip to content

Flaky Go tests on GitHub Actions.
Set up, retried right, quarantined.

On GitHub Actions, Go tests reach Test Engine as a JUnit report: gotestsum writes it, and the mergifyio/gha-mergify-ci action uploads it and decides the job's result with quarantine taken into account. The detail that matters most on GitHub Actions is the test cache, which can skip the very tests you are trying to watch.

By Julien Danjou, Co-founder & CEO of MergifyUpdated

Setup

Run Go with Test Engine on GitHub Actions

Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:

.github/workflows/ci.yml: Write junit.xml with gotestsum, then upload it with the Mergify action, which sets the job's result.
name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-go@v7
        with:
          go-version: stable
      - name: Run tests
        id: tests
        continue-on-error: true
        run: go run gotest.tools/gotestsum@v1.13.0 --junitfile junit.xml -- -count=1 ./...
      - name: Mergify CI Upload
        if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
        uses: mergifyio/gha-mergify-ci@v27
        with:
          token: ${{ secrets.MERGIFY_TOKEN }}
          report_path: junit.xml
          test_step_outcome: ${{ steps.tests.outcome }}
      - name: Fail when tests failed and no report was uploaded
        if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
        run: exit 1
  • continue-on-error: true on the test step is what lets quarantine work: the upload step decides whether the job passes, ignoring failures of quarantined tests. test_step_outcome tells it when the test run itself crashed, so a panic that left a clean report does not pass for green.
  • Pull requests from forks and from Dependabot get no MERGIFY_TOKEN. HAS_MERGIFY_TOKEN skips the upload there, and the last step fails the job when tests failed and nothing was uploaded, or continue-on-error would let those failures through.
  • Recent gotestsum releases need a recent Go (v1.13.0 requires Go 1.24). Pin an older release if your module targets an older Go.
  • Feed the converter go test's standard output only. Merging stderr turns linker warnings into failing tests.

Full reference: Go on docs.mergify.com.

Retries

Re-run failed jobs versus retrying the test

Re-run failed jobs recompiles every package and reruns every test for one that failed, and with the default setup-go cache it may not even rerun the test you care about (see below).

Go has no retry of its own. gotestsum --rerun-fails adds one, and like every blanket retry it reports the passing attempt, so the flake is absorbed.

With Test Engine, every upload adds to each test's history on the default branch and on pull requests, which is how a test that passes and fails on the same commit gets flagged. Once a flaky test is quarantined, the upload step ignores its failure when it sets the job's result, and the test keeps running and reporting until it is healthy again.

Runners

Why Go tests flake on GitHub Actions

  1. The test cache skips the flaky test

    actions/setup-go caches the build cache by default, and go test reuses cached results for packages whose inputs did not change. A flaky test in an untouched package prints (cached) and does not run at all, so it cannot fail, and it cannot report either. -count=1 turns result caching off, which is what you want on the runs that feed flaky detection.

  2. -race on two cores

    The race detector multiplies execution time by 2 to 20 and memory by 5 to 10, per the Go documentation. On a private-repository runner with 2 vCPUs and 8 GB, deadlines written as time.After(100 * time.Millisecond) that hold on a laptop start to miss. Wait on a condition, not on a duration.

    // testify: wait for the condition, not for a fixed delay
    require.Eventually(t, func() bool { return srv.Ready() }, 5*time.Second, 10*time.Millisecond)
  3. t.Parallel() at the runner's width

    -parallel defaults to GOMAXPROCS, which is the runner's CPU count: 2 on a private runner, 4 on a public one, often 10 or more on a laptop. Parallel tests that share a port, a temp directory or a package-level variable interleave differently on each, so a race that never shows locally shows on CI, and the other way around.

The missing layer

GitHub Actions doesn't know what a test is.

GitHub Actions tells you a job failed. Test Engine tells you which Go test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the Go guide for the patterns behind flaky Go suites on any CI: t.Parallel() races, goroutine leaks, and nondeterministic map iteration.

Frequently asked questions

Why do my Go tests pass locally and fail on GitHub Actions?
Usually the runner's width and speed: GOMAXPROCS and -parallel follow its 2 or 4 vCPUs, and -race slows it down further. Reproduce with GOMAXPROCS=2 go test -race -count=1 on the failing package.
Why does a flaky Go test never show up in my reports?
If the package did not change, go test can serve its result from the cache restored by setup-go and print (cached) instead of running it. Run with -count=1 so every test runs and reports.
Why does the upload step need test_step_outcome?
With continue-on-error on the test step, the upload step decides the job's result. test_step_outcome tells it that go test failed, so a crash that left a partial but clean junit.xml fails the job instead of passing it.

Catch the flaky Go tests in your GitHub Actions runs.

Mergify merges more than half a million pull requests a month without breaking main.