Skip to content

Flaky Playwright tests on GitHub Actions.
Set up, retried right, quarantined.

On GitHub Actions, Playwright takes two code changes and one secret: wrap the config with withMergify, import test and expect from @mergifyio/playwright, and give the test step a MERGIFY_TOKEN. The reporter uploads every result and the imported test applies quarantine, so a quarantined test that times out does not turn the job red.

By Julien Danjou, Co-founder & CEO of MergifyUpdated

Setup

Run Playwright with Test Engine on GitHub Actions

Create a Mergify application key with the ci scope and store it as the MERGIFY_TOKEN secret. Then:

playwright.config.js: Wrap the Playwright config with withMergify, which adds the Mergify reporter next to yours.
import { defineConfig, devices } from "@playwright/test";
import { withMergify } from "@mergifyio/playwright";

export default withMergify(
  defineConfig({
    testDir: "./tests",
    reporter: process.env.CI ? "github" : "list",
    projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }],
  }),
);
tests/login.spec.ts: Import test and expect from @mergifyio/playwright in every test and fixture file, which is what applies quarantine.
import { expect, test } from "@mergifyio/playwright";

test("logs in", async ({ page }) => {
  await page.goto("/login");
  await expect(page.getByRole("heading", { name: "Sign in" })).toBeVisible();
});
.github/workflows/ci.yml: Pass the MERGIFY_TOKEN secret to the step that runs Playwright.
name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run tests
        env:
          MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }}
        run: npx playwright test
  • Both changes are required. With the config wrapped but test still imported from @playwright/test, the quarantine list is fetched and never applied, and quarantined tests keep failing the job.
  • Do not pass --reporter on the command line: it replaces the configured reporters, the Mergify one included.
  • Running several browser projects? Set MERGIFY_PLAYWRIGHT_INCLUDE_PROJECT_IN_TEST_NAME: "true" on the step, so a test that fails only on WebKit is tracked as broken on WebKit instead of flaky everywhere.
  • @playwright/test 1.56 or later and Node.js 22 or later in the job.
  • No continue-on-error: quarantine changes the expected status of the test inside the run, so Playwright's exit code is already the right one.

Full reference: Playwright on docs.mergify.com.

Retries

Re-run failed jobs versus retrying the test

Re-run failed jobs is expensive for Playwright: every attempt reinstalls the browsers and their system packages before a single test runs, then replays the whole suite.

The config npm init playwright generates sets retries: process.env.CI ? 2 : 0. Playwright then labels a test that failed and passed on retry as flaky in its HTML report, which is an artifact of that run, and exits 0. The next run starts with no memory of it, and nobody downloads the artifact of a green job.

Test Engine keeps that verdict across runs. Every result reaches it, retries included, so a test Playwright quietly retried to green shows up as flaky in Test Engine, and once it is quarantined, its failures stop blocking merges while it keeps running.

Runners

Why Playwright tests flake on GitHub Actions

  1. One worker on CI, many on your laptop

    The generated config also sets workers: process.env.CI ? 1 : undefined. Locally your tests run in parallel against the same app; on GitHub Actions they run one after the other. A test that relies on data another test created, or on being alone on the server, behaves differently in each. Raising workers on a 2-vCPU private runner does not help either: browsers starve each other and auto-wait timeouts start to fire.

  2. Screenshots taken on another system

    toHaveScreenshot() baselines generated on macOS do not match pixels rendered on Ubuntu, and small antialiasing differences can pass on one attempt and fail on the next. Generate and compare baselines in the same Linux image Playwright publishes, on CI and locally.

    jobs:
      e2e:
        runs-on: ubuntu-latest
        # the tag matches your @playwright/test version
        container: mcr.microsoft.com/playwright:v1.64.1-noble
  3. Shards that change with every pull request

    --shard=1/4 across matrix legs splits tests by file. Adding tests moves others to another shard, so ordering problems surface on pull requests that never touched the failing test. Name each leg after its environment, never its shard: a test can change shard, and a name per shard would split its history.

The missing layer

GitHub Actions doesn't know what a test is.

GitHub Actions tells you a job failed. Test Engine tells you which Playwright test failed, whether it failed on the same commit before, and whether it should block the merge. Read what Test Engine adds to GitHub Actions, or the Playwright guide for the patterns behind flaky Playwright suites on any CI: auto-waiting versus explicit waits, trace-level debugging, and retry config.

Frequently asked questions

Why are my Playwright tests flaky only on GitHub Actions?
Most of the time it is the CI branch of the generated config: one worker and two retries on CI, all workers and no retries locally. Add a runner with 2 vCPUs on private repositories and screenshots rendered on Ubuntu instead of macOS. Run locally with --workers=1 and the same container image to reproduce.
Should I keep retries in playwright.config.ts?
Keep them if a red job costs you more than a slow one, but do not treat a retried pass as a pass. Test Engine records the failed attempt, so the test is flagged as flaky even when the job is green.
Does sharding break flaky detection?
No, as long as the shard index stays out of the job name. Shards run different tests, so they can share one name; a name per shard would split a test's history every time it moves.

Catch the flaky Playwright tests in your GitHub Actions runs.

Mergify merges more than half a million pull requests a month without breaking main.