Flaky Go tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, Go tests reach Test Engine as a JUnit report: gotestsum writes it, and the mergifyio/gha-mergify-ci action uploads it and decides the job's result with quarantine taken into account. The detail that matters most on GitHub Actions is the test cache, which can skip the very tests you are trying to watch.
Setup
Run Go with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
.github/workflows/ci.yml: Write junit.xml with gotestsum, then upload it with the Mergify action, which sets the job's result.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
env:
HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
steps:
- uses: actions/checkout@v7
- uses: actions/setup-go@v7
with:
go-version: stable
- name: Run tests
id: tests
continue-on-error: true
run: go run gotest.tools/gotestsum@v1.13.0 --junitfile junit.xml -- -count=1 ./...
- name: Mergify CI Upload
if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
uses: mergifyio/gha-mergify-ci@v27
with:
token: ${{ secrets.MERGIFY_TOKEN }}
report_path: junit.xml
test_step_outcome: ${{ steps.tests.outcome }}
- name: Fail when tests failed and no report was uploaded
if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
run: exit 1continue-on-error: trueon the test step is what lets quarantine work: the upload step decides whether the job passes, ignoring failures of quarantined tests.test_step_outcometells it when the test run itself crashed, so a panic that left a clean report does not pass for green.- Pull requests from forks and from Dependabot get no
MERGIFY_TOKEN.HAS_MERGIFY_TOKENskips the upload there, and the last step fails the job when tests failed and nothing was uploaded, orcontinue-on-errorwould let those failures through. - Recent
gotestsumreleases need a recent Go (v1.13.0 requires Go 1.24). Pin an older release if your module targets an older Go. - Feed the converter
go test's standard output only. Merging stderr turns linker warnings into failing tests.
Full reference: Go on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs recompiles every package and reruns every test for one that failed, and with the default setup-go cache it may not even rerun the test you care about (see below).
Go has no retry of its own. gotestsum --rerun-fails adds one, and like every blanket retry it reports the passing attempt, so the flake is absorbed.
With Test Engine, every upload adds to each test's history on the default branch and on pull requests, which is how a test that passes and fails on the same commit gets flagged. Once a flaky test is quarantined, the upload step ignores its failure when it sets the job's result, and the test keeps running and reporting until it is healthy again.
Runners
Why Go tests flake on GitHub Actions
The test cache skips the flaky test
actions/setup-gocaches the build cache by default, andgo testreuses cached results for packages whose inputs did not change. A flaky test in an untouched package prints(cached)and does not run at all, so it cannot fail, and it cannot report either.-count=1turns result caching off, which is what you want on the runs that feed flaky detection.-race on two cores
The race detector multiplies execution time by 2 to 20 and memory by 5 to 10, per the Go documentation. On a private-repository runner with 2 vCPUs and 8 GB, deadlines written as
time.After(100 * time.Millisecond)that hold on a laptop start to miss. Wait on a condition, not on a duration.// testify: wait for the condition, not for a fixed delay require.Eventually(t, func() bool { return srv.Ready() }, 5*time.Second, 10*time.Millisecond)t.Parallel() at the runner's width
-paralleldefaults toGOMAXPROCS, which is the runner's CPU count: 2 on a private runner, 4 on a public one, often 10 or more on a laptop. Parallel tests that share a port, a temp directory or a package-level variable interleave differently on each, so a race that never shows locally shows on CI, and the other way around.
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which Go test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the Go guide for the patterns behind flaky Go suites on any CI: t.Parallel() races, goroutine leaks, and nondeterministic map iteration.
Frequently asked questions
- Why do my Go tests pass locally and fail on GitHub Actions?
- Usually the runner's width and speed: GOMAXPROCS and -parallel follow its 2 or 4 vCPUs, and -race slows it down further. Reproduce with GOMAXPROCS=2 go test -race -count=1 on the failing package.
- Why does a flaky Go test never show up in my reports?
- If the package did not change, go test can serve its result from the cache restored by setup-go and print (cached) instead of running it. Run with -count=1 so every test runs and reports.
- Why does the upload step need test_step_outcome?
- With continue-on-error on the test step, the upload step decides the job's result. test_step_outcome tells it that go test failed, so a crash that left a partial but clean junit.xml fails the job instead of passing it.