Flaky Cypress tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, Cypress reaches Test Engine as JUnit XML. mocha-junit-reporter writes one report per spec file, cypress-multi-reporters keeps the usual spec output in the log next to it, and the mergifyio/gha-mergify-ci action uploads the reports and sets the job's result with quarantine taken into account. Two details decide whether those reports tell the truth: your app has to answer before cypress run starts, and Cypress's own retries have to stay off, or a failed attempt is written to the report as a pass.
Setup
Run Cypress with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
cypress.config.js: Point Cypress at cypress-multi-reporters and its config file. Without it, Cypress keeps its default spec reporter and writes no JUnit file.const { defineConfig } = require("cypress");
module.exports = defineConfig({
reporter: "cypress-multi-reporters",
reporterOptions: {
configFile: "reporter-config.json",
},
e2e: {
baseUrl: "http://127.0.0.1:8080",
},
});reporter-config.json: Keep the spec reporter for the log and write one JUnit file per spec file, named by hash so no spec overwrites another's report.{
"reporterEnabled": "spec, mocha-junit-reporter",
"mochaJunitReporterReporterOptions": {
"mochaFile": "cypress/results/junit-[hash].xml"
}
}package.json: Start the app, wait until it answers with a 200, then run Cypress in the runner's Chrome. Install the tools with npm install --save-dev cypress cypress-multi-reporters mocha-junit-reporter start-server-and-test.{
"scripts": {
"start": "http-server public -p 8080 -s",
"cy:run": "cypress run --browser chrome",
"test:cypress": "start-server-and-test start http://127.0.0.1:8080 cy:run"
}
}.github/workflows/ci.yml: Run Cypress with continue-on-error, then upload every report with the Mergify action, which sets the job's result.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
env:
HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }}
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: 24
cache: npm
- run: npm ci
- name: Run Cypress tests
id: tests
continue-on-error: true
run: npm run test:cypress
- name: Mergify CI Upload
if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true'
uses: mergifyio/gha-mergify-ci@v28
with:
token: ${{ secrets.MERGIFY_TOKEN }}
report_path: cypress/results/junit*.xml
test_step_outcome: ${{ steps.tests.outcome }}
- name: Fail when tests failed and no report was uploaded
if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true'
run: exit 1continue-on-error: trueon the Cypress step is what lets quarantine work: the upload step decides whether the job passes and ignores failures of quarantined tests.test_step_outcometells it when the run itself failed. With one report per spec file, a run that stopped halfway still leaves clean reports for the specs that finished, and those alone would read as green.- Keep
[hash]inmochaFile. Cypress runs each spec file on its own, so a fixed file name is overwritten by each spec and only the last spec's results would reach Test Engine.report_path: cypress/results/junit*.xmluploads all of them. - Pass
--browser chrome. Electron is still Cypress's default browser and is deprecated as a test browser, and the Ubuntu runner images already ship Chrome, Firefox and Edge. - Pull requests from forks and from Dependabot get no
MERGIFY_TOKEN.HAS_MERGIFY_TOKENskips the upload on those runs, and the last step fails the job when Cypress failed and nothing was uploaded, sincecontinue-on-errorwould otherwise let those failures through. - Running the suite across browsers in a matrix? Every leg reports the same
GITHUB_JOB, so give the upload step ajob_nameper leg, such asjob_name: test (${{ matrix.browser }}). Otherwise a spec that only fails in Firefox looks flaky instead of broken. cache: npmin setup-node caches npm's package data, and the Cypress binary lives elsewhere, in~/.cache/Cypress. Sonpm cidownloads the binary again on every run. Cypress recommends caching that folder; cypress-io/github-action does it for you.
Full reference: Cypress on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs replays the whole job for one spec that may have been unlucky: npm ci, the Cypress binary download, the app's start, Chrome's start, then every spec file, the ones that passed included. If you also record to Cypress Cloud, Cypress's GitHub Actions guide lists re-runs that pass with empty tests among its common problems, and passes GITHUB_TOKEN to the step so a re-run is identified as one.
Cypress's own test retries are off by default, retries: { runMode: 0, openMode: 0 } per the configuration reference. Set runMode: 2 and a failing test is retried in place up to twice, beforeEach and afterEach hooks included, and the first attempt that passes is its result. We ran a test that fails on its first attempt only through this setup on GitHub Actions: mocha-junit-reporter wrote it as a plain passing <testcase>, with no trace of the failed attempt. The only evidence left on the runner was a screenshot whose name ends in (failed).png.
That pass is all Test Engine would receive, so leave retries at 0 on the runs you upload. Every failure then reaches Test Engine, a test that fails and passes on the same commit is flagged as flaky, and once it is quarantined, the upload step ignores its failure when it sets the job's result while the spec keeps running and reporting. If you want Cypress to retry anyway, its experimental detect-flake-but-always-fail strategy (Cypress 13.4 and later) keeps the failure: in the same experiment, the test came out of the report as failed, with its first attempt's error.
Runners
Why Cypress tests flake on GitHub Actions
Chrome, Cypress and your app on two vCPUs
A private-repository
ubuntu-latestrunner has 2 vCPUs and 8 GB, where a public repository gets 4 vCPUs and 16 GB. Headless Chrome, the Cypress process and the server you started share them, and most DOM commands still get the same 4-seconddefaultCommandTimeoutthey had on your laptop. The assertion that waits on the slowest response is the first to cross it, on the runs where the runner is busiest. Wait for the request withcy.interceptandcy.waitwhere you can; where CI is just slower, Cypress reads the timeout from the environment, so you can raise it for CI alone.- name: Run Cypress tests id: tests continue-on-error: true env: CYPRESS_DEFAULT_COMMAND_TIMEOUT: 8000 run: npm run test:cypressAn app that is started but not answering
Your app runs inside the same job as Cypress, and a server whose process has started may not accept connections yet.
npm start & npx cypress runis the race Cypress's CI guide tells you not to write: on a cold runner the firstcy.visitlands before the port is open, the first spec fails, and the re-run passes because the server won that time.start-server-and-test, as in the setup above, waits for a 200 before it starts Cypress; cypress-io/github-action does the same with itswait-oninput.A Chrome version that changes between attempts
GitHub updates the browsers in its runner images, and Cypress notes that while a new image is rolling out, the "Set up job" phase can land on the old or the new one. Two attempts on the same commit can then run different Chrome versions, and a failure that only happens on one of them looks flaky. A
cypress/browsersimage with a long-form tag pins Node.js and each browser to an exact version.jobs: test: runs-on: ubuntu-latest # Node.js, Chrome, Firefox and Edge versions fixed by the tag container: image: cypress/browsers:node-24.22.1-chrome-156.0.8078.73-1-ff-159.0-edge-156.0.4314.50-1Spec files split by hand across matrix legs
--parallelrequires recording to Cypress Cloud, so without it, teams split spec files across matrix legs with--spec. When each leg starts its own app and database, a spec that relied on data another spec left behind passes while both land on the same leg, and starts failing on the pull request that adds a file and moves it. Keep the leg out of the upload step'sjob_name: the legs run different specs, and a name per leg would split a spec's history each time it moves.
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which Cypress test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the Cypress guide for the patterns behind flaky Cypress suites on any CI: cy.wait() races, network stubbing, and retry-ability gotchas.
Frequently asked questions
- Why do my Cypress tests pass locally but fail on GitHub Actions?
- On GitHub Actions, cypress run is headless, on 2 vCPUs shared with your app when the repository is private, against a server that may not be answering yet, in whichever Chrome the runner image ships. Locally you likely run cypress open, headed, on a faster machine. Reproduce with npx cypress run --browser chrome, headless like CI, against a server started the way CI starts it, and add --headed --no-exit to watch the run and inspect the page once it ends.
- Should I turn on retries in cypress.config.js?
- Not on the runs you upload. A test that fails and then passes on its retry is written to the JUnit report as a plain pass, so Test Engine never sees the failure and cannot flag it. Leave retries at 0 and let quarantine keep the known flaky specs from blocking merges. If you keep retries, the experimental detect-flake-but-always-fail strategy reports such a test as failed.
- Can I keep cypress-io/github-action?
- Yes. Give the action step id: tests and continue-on-error: true, and keep the Mergify upload step after it. The reporters are set in cypress.config.js, so they apply however Cypress is started, and the action's start and wait-on inputs replace start-server-and-test. It also caches the Cypress binary, which setup-node does not.