Flaky Playwright tests on GitHub Actions.
Set up, retried right, quarantined.
On GitHub Actions, Playwright takes two code changes and one secret: wrap the config with withMergify, import test and expect from @mergifyio/playwright, and give the test step a MERGIFY_TOKEN. The reporter uploads every result and the imported test applies quarantine, so a quarantined test that times out does not turn the job red.
Setup
Run Playwright with Test Engine on GitHub Actions
Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:
playwright.config.js: Wrap the Playwright config with withMergify, which adds the Mergify reporter next to yours.import { defineConfig, devices } from "@playwright/test";
import { withMergify } from "@mergifyio/playwright";
export default withMergify(
defineConfig({
testDir: "./tests",
reporter: process.env.CI ? "github" : "list",
projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }],
}),
);tests/login.spec.ts: Import test and expect from @mergifyio/playwright in every test and fixture file, which is what applies quarantine.import { expect, test } from "@mergifyio/playwright";
test("logs in", async ({ page }) => {
await page.goto("/login");
await expect(page.getByRole("heading", { name: "Sign in" })).toBeVisible();
});.github/workflows/ci.yml: Pass the MERGIFY_TOKEN secret to the step that runs Playwright.name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: 24
cache: npm
- run: npm ci
- run: npx playwright install --with-deps chromium
- name: Run tests
env:
MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
run: npx playwright test- Both changes are required. With the config wrapped but
teststill imported from@playwright/test, the quarantine list is fetched and never applied, and quarantined tests keep failing the job. - Do not pass
--reporteron the command line: it replaces the configured reporters, the Mergify one included. - Running several browser projects? Set
MERGIFY_PLAYWRIGHT_INCLUDE_PROJECT_IN_TEST_NAME: "true"on the step, so a test that fails only on WebKit is tracked as broken on WebKit instead of flaky everywhere. @playwright/test1.56 or later and Node.js 22 or later in the job.- No
continue-on-error: quarantine changes the expected status of the test inside the run, so Playwright's exit code is already the right one.
Full reference: Playwright on docs.mergify.com.
Retries
Re-run failed jobs versus retrying the test
Re-run failed jobs is expensive for Playwright: every attempt reinstalls the browsers and their system packages before a single test runs, then replays the whole suite.
The config npm init playwright generates sets retries: process.env.CI ? 2 : 0. Playwright then labels a test that failed and passed on retry as flaky in its HTML report, which is an artifact of that run, and exits 0. The next run starts with no memory of it, and nobody downloads the artifact of a green job.
Test Engine keeps that verdict across runs. Every result reaches it, retries included, so a test Playwright quietly retried to green shows up as flaky in Test Engine, and once it is quarantined, its failures stop blocking merges while it keeps running.
Runners
Why Playwright tests flake on GitHub Actions
One worker on CI, many on your laptop
The generated config also sets
workers: process.env.CI ? 1 : undefined. Locally your tests run in parallel against the same app; on GitHub Actions they run one after the other. A test that relies on data another test created, or on being alone on the server, behaves differently in each. Raising workers on a 2-vCPU private runner does not help either: browsers starve each other and auto-wait timeouts start to fire.Screenshots taken on another system
toHaveScreenshot()baselines generated on macOS do not match pixels rendered on Ubuntu, and small antialiasing differences can pass on one attempt and fail on the next. Generate and compare baselines in the same Linux image Playwright publishes, on CI and locally.jobs: e2e: runs-on: ubuntu-latest # the tag matches your @playwright/test version container: mcr.microsoft.com/playwright:v1.64.1-nobleShards that change with every pull request
--shard=1/4across matrix legs splits tests by file. Adding tests moves others to another shard, so ordering problems surface on pull requests that never touched the failing test. Name each leg after its environment, never its shard: a test can change shard, and a name per shard would split its history.
The missing layer
GitHub Actions doesn't know what a test is.
GitHub Actions tells you a job failed. Test Engine tells you which Playwright test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the Playwright guide for the patterns behind flaky Playwright suites on any CI: auto-waiting versus explicit waits, trace-level debugging, and retry config.
Frequently asked questions
- Why are my Playwright tests flaky only on GitHub Actions?
- Most of the time it is the CI branch of the generated config: one worker and two retries on CI, all workers and no retries locally. Add a runner with 2 vCPUs on private repositories and screenshots rendered on Ubuntu instead of macOS. Run locally with --workers=1 and the same container image to reproduce.
- Should I keep retries in playwright.config.ts?
- Keep them if a red job costs you more than a slow one, but do not treat a retried pass as a pass. Test Engine records the failed attempt, so the test is flagged as flaky even when the job is green.
- Does sharding break flaky detection?
- No, as long as the shard index stays out of the job name. Shards run different tests, so they can share one name; a name per shard would split a test's history every time it moves.