# Mergify: full text of the product, pricing, comparison, guide and learn pages > Every page below is also served on its own at its URL plus `.md`. Index of the whole site, blog posts and customer stories included: https://mergify.com/llms.txt --- Source: https://mergify.com/product/ci-insights CI Insights # Your CI is a black box. Open it up. How many hours did your team waste on flaky CI jobs this week? CI Insights tells you exactly which jobs are unreliable, retries them automatically, and gives you the data to fix root causes. Teams cut CI debugging time by 60%. [Get started](https://dashboard.mergify.com/ci-insights)[Read the docs](https://docs.mergify.com/ci-insights/) Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama ## Before CI Insights Lack of visibility into CI slowdowns & failures 15 manual CI job re-runs per week, per engineer Engineers guess which job flaked Debugging buried in logs ## After CI Insights Instant visibility into slowdowns & failure alerts Less than 3 manual CI job re-runs in total Flaky jobs are flagged and tracked automatically PR comments summarize what broke + where ## Stop guessing why CI is red. See exactly what broke and why. ### From high level CI performance, to detailed job execution insights See overall performance and understand the root cause of job failures and slowdowns. [Get started](https://dashboard.mergify.com/ci-insights) #### Understand Job health See overall performance of your jobs. From the number of executions, durations, and failure rates, to the overall health score and flakiness of job runs. #### Spot slowdowns and regressions Track job durations over time and catch regressions before they compound. Know when a job starts taking longer and why. #### PR Summary Reports Get immediate summary reports of any CI issues posted as comments on your PRs in GitHub. #### Get notified about issues in real-time Using our Slack integration you'll be able to get notified about any desired job outcome across particular repositories, branches, and jobs. > Before CI Insights, we were wasting hours debugging CI issues. We now cut triage time in half within a week. Liam Hartman Platform Engineer at Vertex ## Want to see what CI Insights finds in your pipelines? 5-minute setup. Works with GitHub Actions, Jenkins, Buildkite, and soon CircleCI. [Book a discovery call](https://mergify.com/discovery) ## Auto-retry flaky jobs. Track every CI issue. Retry jobs without lifting a finger and make sure you easily keep track of every CI issue you encounter. [Get started](https://dashboard.mergify.com/ci-insights) ### Automatic job retries Automatically re-run CI jobs without restarting your entire pipeline. Use fine-grained filters to target specific jobs. Save time, reduce CI noise, and keep your builds flowing. ### CI Issue Tracking Running into a CI issue? Mark it and track your CI issues directly in Mergify to solve it with your team and sync it with your favorite Issue Tracking tool. > Tracking down flaky jobs and failures took hours. Our mean time to resolution dropped by 60% when we started using CI Insights. Liam Hartman Platform Engineer at Vertex ## Self-hosted runner waste is real. Find it. Fix it. ### Every runner in your fleet, scored on the metrics that actually matter Most teams know self-hosted runners are cheaper than GitHub-hosted, but they can't prove which ones are pulling their weight. CI Insights tracks every runner and shows you the ones costing you money for nothing. [Get started](https://dashboard.mergify.com/ci-insights) We don't sell runners, so we have no reason to push you toward any provider. #### Fleet performance at a glance Success rate, throughput, runtime, and queue time over any window, with week-over-week comparison built in. Stop scrolling through dashboards to answer "is CI getting worse?" #### Per-runner deep stats p95 queue time, p99 duration, failure rate, and total cost for every runner in the fleet. Stop arguing about which node is slow and start fixing it. #### Degraded runner detection Mergify flags the runners that are limping along before they tank your throughput. You spot the bad nodes without scrolling through logs or graphs. #### Cost behind every job Every minute and every dollar tied back to the runner that spent it. The data finance has been asking for, in a format they'll accept. JenkinsAvailable now GitHub ActionsAvailable now BuildkiteAvailable now CircleCIComing soon ## One dashboard. Every CI provider. GitHub Actions for half your repos, Jenkins for the rest, CircleCI for that one team that refuses to migrate. CI Insights pulls job health from all of them into a single view. Flaky detection and auto-retry work the same way no matter which CI ran the job. [Get started](https://dashboard.mergify.com/ci-insights) ## CI problems that get worse the longer you ignore them Most teams don't realize how much CI costs them until they measure it. ### No visibility into CI cost Leadership can't explain where CI time and compute go. Costs rise but nobody can point to why. ### Engineers waste hours on reruns 15 manual re-runs per week per engineer is normal. Most of those are flaky jobs that could be retried automatically. ### Flaky jobs erode trust When CI results are unreliable, engineers stop paying attention. Reviews slow down, broken code slips through. ### CI health degrades silently Without tracking, you can't tell if CI is getting worse over time. By the time it's obvious, the backlog of broken jobs is massive. ## Teams that merge more and break main less. [ ### 40% faster CI runs, fewer flaky failures Read story → AheadComputing 100 engineers · Semiconductor Before Mergify: GitHub's merge queue ](https://mergify.com/customers/aheadcomputing) [ ### Zero breaking changes on main since adopting Mergify Read story → Zama 30 engineers · Blockchain / Cryptography ](https://mergify.com/customers/zama) [ ### 2x throughput with no CI expansion Read story → Berkshire Grey 100 engineers · Robotics & Artificial Intelligence Before Mergify: Bulldozer ](https://mergify.com/customers/berkshiregrey) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## Slash your CI debugging time with CI Insights Calculate what flaky tests cost your team each month, with a breakdown you can share with leadership. [See your ROI](https://mergify.com/calculator)[Read the docs](https://docs.mergify.com/ci-insights/) --- Source: https://mergify.com/product/merge-queue Merge Queue· for teams outgrowing GitHub's # Main stays green. Your team stays fast. Batches that cut CI runs, a lane for each part of your monorepo, and flaky tests that stop ejecting good pull requests. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/merge-queue/) - [Batching1 CI run](#batching) - [Fail Firstfastest bisection](#fail-first) - [Anti-flakeflakes don't eject](#anti-flake) - [Lanesno waiting](#parallel-lanes) - [Prioritieshotfix first](#priorities) - [Stacksland in order](#stacked-pull-requests) - [Freezespause merges](#freezes-and-dependencies) Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama 2x merge throughput Cerebras doubled merge throughput while cutting down on its costly 3-hour CI runs. [Read the story](https://mergify.com/customers/cerebras) The problem ## Your team ships faster than main can take it. Agents open more pull requests than ever, and flaky tests haven't gone anywhere. They all land on the same queue. SituationGitHub's merge queueMergify More pull requestsagents open them around the clock GitHub's merge queue: **a CI run for each**CI load grows with every PR Mergify: **one run per batch**many PRs, tested together Flaky testsa red run that isn't a bug GitHub's merge queue: **the PR is ejected**and the runs behind it restart Mergify: **it still merges**a later passing batch vouches for it Unrelated changesa CSS fix behind a DB migration GitHub's merge queue: **one line for all**the fix restarts when the migration fails Mergify: **a lane per scope**the fix never waits Want every difference? [Compare with GitHub's merge queue](https://mergify.com/compare/github-merge-queue) ## How Mergify keeps your queue moving Batching ### One CI run for many pull requests Set a `batch_size` and Mergify groups the pull requests that are ready into one batch, tests them together the way they will land on main, and merges them in one go. Ten pull requests can cost one CI run instead of ten. .mergify.yml ``` queue_rules: - name: default batch_size: 4 ``` Pair it with two-step CI: light checks on every push to a pull request, the full suite only once, in the queue. Long suites stop running on code that never merges. [Read about batches](https://docs.mergify.com/merge-queue/batches/) Fail First bisection ### The fastest bisection on the market When a batch fails, the queue splits it to find the pull request responsible. With Fail First, each half starts with the tests that broke the batch, so the guilty half fails in seconds instead of after a full run. The innocent pull requests keep going and merge, and the culprit leaves the queue. [Read about batch failures](https://docs.mergify.com/merge-queue/batches/) Anti-flake ### A flaky failure doesn't eject anyone Turn on anti-flake protection and a passing batch vouches for an earlier failing one. If a later batch containing the same changes passes, the earlier failure is treated as transient and the pull requests merge. A real bug would fail the bigger batch too, so only flaky failures get through. Quarantine flaky tests across your repositories with [Test Engine](https://mergify.com/product/test-engine). [Read about anti-flake protection](https://docs.mergify.com/merge-queue/batches/) Parallel lanes ### Unrelated changes stop waiting on each other Each scope gets its own lane, so a CSS change never waits for a database migration. Mergify can match file paths itself, or take the scopes your build tool computes. File paths ``` scopes: source: files: payments: include: [payments/**] web: include: [web/**] ``` Build graph ``` scopes: source: manual: ``` Your CI runs bazel query, nx affected, Turborepo or Pants and sends the scopes with mergify ci scopes-send or the API. A pull request that spans two scopes waits for room in both. One that touches everything can be flagged to run on its own. [Read about scopes](https://docs.mergify.com/merge-queue/scopes/) ## Control when code lands. GitHub's queue can only jump a pull request to the front, and every run behind it starts over. Mergify lets you set priorities, freeze merges and wait on dependencies. ### Priorities Priority levels by label or rule. A hotfix goes ahead of the feature work queued before it. [Priorities](https://docs.mergify.com/merge-queue/priority/) ### Freezes Freeze merges for a release or an incident, on a schedule or on the spot. The queue keeps testing, and a hotfix exception still goes through. [Code freeze guide](https://mergify.com/guides/merge-queue-code-freeze) ### Depends-On and Merge-After A `Depends-On` line holds a pull request until the ones it names have merged, across repositories. `Merge-After` holds it until a set time. [Depends-On guide](https://mergify.com/guides/manage-dependent-prs-depends-on) ### Stacked pull requests Comment @mergifyio queue on the top pull request and the whole stack enters the queue at once. It travels in one batch and lands in order, lowest first. [Stacks](https://docs.mergify.com/merge-queue/stacks/) Moved over from GitHub's merge queue - - - > "Mergify gives us the control and visibility we never had with GitHub's queue. It's become our CI command center." Ben Holtzman, Senior Design Enablement Engineer, [AheadComputing](https://mergify.com/customers/aheadcomputing) ### Switching from GitHub's queue Your CI, workflows and required checks stay as they are. About five minutes for the first repository. 1. 1 · Today Branch ruleset · main Require merge queue GitHub's queue runs every merge. 3. 2 · Turn off the rule Branch ruleset · main Require merge queue Install the Mergify GitHub App on the repository. 5. 3 · Queue with Mergify you commented now @mergifyio queue mergify bot commented now Queued in the merge queue Batch · 3 PRschecking A .mergify.yml comes later, for batches or lanes. Going back takes the same two steps: turn the rule back on and uninstall the App. [Compare with GitHub's merge queue](https://mergify.com/compare/github-merge-queue) ## How it fits your setup Mergify sits between your approved pull requests and main. Your CI keeps running the way it does today. Source control GitHub.com Enterprise Server Your CI, unchanged GitHub Actions BuildkiteJenkinsCircleCI Azure Pipelines Mergify Merge queue approved PRs → tested batches → main Notifications Slack Checks and comments Monitoring and automation Datadog REST API CLI ## Safe to put in the merge path ✓ SOC 2 Type II Report, audit details and sub-processors on the trust portal. [Trust portal](https://trust.mergify.com/) ✓ Roles from GitHub Sign-in goes through GitHub and roles are inherited from it. Every App permission is documented. [Permissions](https://docs.mergify.com/security/) ✓ On-premise Run Mergify Enterprise Server inside your network, with GitHub.com or GitHub Enterprise Server. [Enterprise Server](https://mergify.com/solutions/enterprise-server) ✓ Statistics and event log Throughput, wait times and failure rates in the dashboard, and an event log of what each pull request went through. [Monitoring](https://docs.mergify.com/merge-queue/monitoring/) ## Teams that merge more and break main less. [ ### Zero breaking changes on main since adopting Mergify Read story → Zama 30 engineers · Blockchain / Cryptography ](https://mergify.com/customers/zama) [ ### 40% faster CI runs, fewer flaky failures Read story → AheadComputing 100 engineers · Semiconductor Before Mergify: GitHub's merge queue ](https://mergify.com/customers/aheadcomputing) [ ### 2x throughput with no CI expansion Read story → Berkshire Grey 100 engineers · Robotics & Artificial Intelligence Before Mergify: Bulldozer ](https://mergify.com/customers/berkshiregrey) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## FAQ ### How is Mergify different from GitHub's native merge queue? GitHub's queue tests each queued pull request in its own merge group, together with the ones ahead of it, and runs several groups in parallel. Mergify adds parallel queues by scope (a frontend PR does not wait behind a backend migration), batches several PRs into a single CI run and splits the batch to find the culprit when it fails, runs two-step CI (lightweight checks on PRs, full suite in the queue), and keeps known flaky tests from ejecting pull requests. [See the full comparison](https://mergify.com/compare/github-merge-queue). ### How long does it take to set up? About five minutes for the first repository, with no configuration file. Install the GitHub App, then comment @mergifyio queue on a pull request and follow it through the queue from the dashboard. Larger setups with parallel queues, custom CI integration, batching, and multi-team rules take longer to design but are still day-one work. ### Can we go back to GitHub's merge queue? Yes. Going back takes the same two steps as coming over: turn the "Require merge queue" rule back on and uninstall the GitHub App. Your CI and GitHub permissions never changed, and the `.mergify.yml` is a plain file you can delete. ### Does Mergify work with GitHub Actions? Yes. Mergify pushes each candidate merge to a temporary queue branch and opens a draft PR for it, so workflows you already trigger on push or pull\_request run there automatically. No workflow rewrite required. ### Can Mergify handle monorepos? Yes. Your CI computes scopes with your build tool and sends them to Mergify, so the lanes track what your monorepo actually is instead of a config file someone has to keep current. Bazel, Nx, Turborepo and Pants are covered, and anything else can push its own scopes through the public API. File patterns are there as the zero-setup option when you have no build tool. Each scope gets its own lane, so a CSS change never waits behind a database migration, and a pull request whose scopes nothing else has touched merges directly without running queue CI at all. ### Do I need to write YAML to use Mergify? No. A working queue takes no configuration file at all: install the Mergify GitHub App and comment @mergifyio queue on a pull request. Mergify updates the PR against the latest main, runs the CI you already have, and merges when it passes. The .mergify.yml file comes in later, when you want parallel queues, batch sizes, priority lanes, monorepo scopes, or automatic queueing of approved PRs. ### What is a merge queue? A merge queue is a system that sits between approved pull requests and the main branch. It tests each PR against the actual state main will have at merge time, then only merges when CI passes. Main stays green even when many engineers ship in parallel. ### How often does a green pull request actually break main? It depends on team size. Across 200,000 merges from 477 organizations, a pull request that passed CI still broke main 0.77% of the time at 2 to 5 engineers (1 in 130) and 12.5% of the time once a repository passed 40 engineers (1 in 8). Private codebases hit it 5.1% of the time against 1.1% for open source. That distance between "CI passed" and "safely merged" is the validation gap, and closing it is what a merge queue does. Method and full data in the [State of Merge Queues 2026](https://mergify.com/reports/state-of-merge-queues-2026). ### Does Mergify work with GitHub stacked pull requests? Yes. A pull request that belongs to a GitHub stack can enter the queue like any other, and Mergify merges the members in order, lowest first, so a stack never lands out of sequence. Mergify reads the stack from GitHub itself, whether you built it in the web UI, with `gh stack`, or from the mobile app, so there is nothing to configure and no marker to add by hand. ### Can I freeze merges during a release or an incident? Yes. Schedule a freeze ahead of time, or start one on the spot, and nothing matching it merges until it lifts. The queue keeps testing pull requests in the meantime, so they land the moment the freeze ends, and you can carve out an exception so a hotfix still goes through. See [how to run a merge queue during a code freeze](https://mergify.com/guides/merge-queue-code-freeze). ### Can a pull request wait for another one to merge first? Yes. Add a `Depends-On` line to its description and Mergify holds it until the pull requests it names have merged, including ones in other repositories. A `Merge-After` line holds it until a given time. See [how to manage dependent PRs with Depends-On](https://mergify.com/guides/manage-dependent-prs-depends-on). ### Does Mergify do anything besides the merge queue? Yes, and every plan includes all of it at no extra cost. [CI Insights](https://mergify.com/product/ci-insights) shows which CI jobs are slow or failing and where your self-hosted runner spend goes, and Test Engine finds unreliable tests. Most teams start with the queue and turn on CI Insights once they want to know what is slowing it down. ### What does Mergify cost? Free for open-source projects and for teams up to 5 users on private repositories, with every feature included and no credit card. Max is $21 per seat per month, or $216 per seat per year, up to 100 users. Enterprise is quoted per seat and adds on-premise deployment, 24/7 support, custom data retention, and a quarterly roadmap review. [See pricing](https://mergify.com/pricing) for the full comparison. Want the full picture? [Read the merge queue guide](https://mergify.com/learn/merge-queue) or [compare with GitHub's merge queue](https://mergify.com/compare/github-merge-queue). Last updated: October 16, 2026 ## Keep main green. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Talk to our team](https://mergify.com/discovery) --- Source: https://mergify.com/product/test-engine Test Engine Formerly Test Insights # Flaky tests block your whole team. Stop them at the pull request. Test Engine is the test layer of Mergify's CI platform, and it plugs into the test frameworks you already use. It catches flaky tests on the pull request before they merge, and quarantines the ones already in your suite so they stop blocking merges. [Get started](https://dashboard.mergify.com/test-engine)[Read the docs](https://docs.mergify.com/test-engine/) Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama ## One flaky test. A full day wasted. Flaky tests don't just fail. They cascade. Here's what a single unreliable test does to your team every day. Most teams have 10, 20, or 40 of these tests. Nobody tracks them. Nobody owns them. The cost compounds silently until CI results mean nothing. Plugs into your test suite ## Native in every test framework Test Engine runs inside the test framework you already use, whatever CI sits underneath. pytest Jest RSpec Vitest Playwright Cypress Go Rust JUnit TestNG PHPUnit Pest NUnit MSTest minitest Results upload from inside the run and quarantine applies there too, so a quarantined failure never touches your exit code. GitHub Actions, Buildkite and Jenkins each have a setup guide. Don't see your framework? [Tell us and we'll probably build it.](https://mergify.com/discovery) The fix ## What Test Engine does with every test run Prevention stops new flaky tests at the pull request. Detection ranks the ones already on your default branch by how much they hurt, and Quarantine keeps their failures out of your merges while someone fixes them. Other tools find a flaky test after it has merged. Test Engine catches it on the pull request, before it reaches your default branch. [Get started](https://dashboard.mergify.com/test-engine) Pillar 1 ### Prevention Catch flaky tests on the pull request, before they merge to main. Test Engine reruns the tests a pull request adds or changes, on the same commit, and flags every one that flips between pass and fail. Each new test shows up as healthy, flaky or broken before anyone clicks merge. Reruns stay inside a CI budget you set, and run through a native plugin such as pytest-mergify or rspec-mergify. No more merging a "green" PR only to discover the test was flaky all along. Pillar 2 ### Detection Detection finds the unhealthy tests already on your default branch and ranks them by impact, the share of their runs that failed, so the ones costing you the most get fixed first. Every test sits under Healthy, Flaky or Broken. With a plugin installed, Test Engine can rerun a suspect test on the default branch to confirm the verdict instead of waiting for its next failure. A test that passes for a week after its last failure goes back to healthy on its own. No more guessing whether a failure is real or just noise. Pillar 3 ### Quarantine Quarantine is a state a test is in. A quarantined test keeps running and reporting, but its failures stop blocking merges or failing the pipeline, on every branch. Turn on auto-quarantine and Test Engine puts a flaky test there by itself once enough runs back the verdict (broken tests too, if you opt in). One red run among thousands of green ones never quarantines anything. Every move in or out of quarantine, yours or Test Engine's, lands in the event log and can post to Slack. No more `@skip` with a TODO nobody reads. > We had 40 known-flaky tests and no system to deal with them. Engineers just re-ran the suite and hoped. Test Engine quarantined the worst offenders on day one, and within a month we actually fixed most of them because we could finally see which ones mattered. Lucas Meier Staff Software Engineer at Norde Part of the CI platform ## Your tests and your merge queue finally talk A standalone flaky-test tool can tell you a test is unreliable. Test Engine sits next to [Merge Queue](https://mergify.com/product/merge-queue) and [CI Insights](https://mergify.com/product/ci-insights), so what it knows about a test changes what the queue does with it. ### A known flake can't kick a PR out of the queue Quarantine applies on every branch, the merge queue's included. When a quarantined test fails during a queue run, the pipeline still passes, so the pull request keeps its place instead of getting dequeued. ### One CI connection for jobs and tests Test Engine runs on the same CI integration as CI Insights. Connect GitHub Actions, Buildkite or Jenkins once, and both read from it. ## Without a test layer, your CI pays the bill Most teams underestimate how much flaky tests cost them until the problem is measurable. ### Developer trust breaks down When tests fail randomly, engineers stop trusting CI. They re-run, skip, or ignore results. Quality slips. ### Merge queues grind to a halt One flaky test blocks a batch of PRs. The whole queue retries. Multiply that across a day and you lose hours of engineering time. ### Debugging is a time sink Is the test broken or flaky? Engineers waste hours answering a question the tool should answer for them. ### The backlog grows silently Without tracking, flaky tests accumulate. By the time it is painful, there are dozens to triage and no data to prioritize them. ## Want to see Test Engine on your test suite? Native in pytest, Jest, RSpec, Vitest, Playwright, Go and every other major test framework. [Book a discovery call](https://mergify.com/discovery) ### Dig deeper by framework How flaky tests fail in each framework, and the fixes that actually hold. [Jest](https://mergify.com/learn/flaky-tests/jest)[pytest](https://mergify.com/learn/flaky-tests/pytest)[Cypress](https://mergify.com/learn/flaky-tests/cypress)[Playwright](https://mergify.com/learn/flaky-tests/playwright)[Vitest](https://mergify.com/learn/flaky-tests/vitest)[RSpec](https://mergify.com/learn/flaky-tests/rspec)[Minitest](https://mergify.com/learn/flaky-tests/minitest)[Go](https://mergify.com/learn/flaky-tests/go)[Rust](https://mergify.com/learn/flaky-tests/rust)[JUnit](https://mergify.com/learn/flaky-tests/junit)[TestNG](https://mergify.com/learn/flaky-tests/testng)[PHPUnit](https://mergify.com/learn/flaky-tests/phpunit)[Pest](https://mergify.com/learn/flaky-tests/pest)[NUnit](https://mergify.com/learn/flaky-tests/nunit)[MSTest](https://mergify.com/learn/flaky-tests/mstest)[All frameworks →](https://mergify.com/learn/flaky-tests) Bring your own agent ## Your AI agent doesn't know which tests are flaky. Now it can. AI coding tools write code and debug failures, but they waste time investigating tests that were already known-flaky. The Mergify CLI gives any agent access to your test health data. No integration needed. Just a CLI call. Terminal $ mergify tests quarantines list --repository myorg/myapp tests/payments/test\_webhook.py::test\_payment\_webhook   branch:     \*   recovered:  no tests/email/test\_delivery.py::test\_email\_delivery   branch:     \*   recovered:  no Your AI agent can now skip these when debugging CI failures. ### One CLI call. Any agent. Cursor, Claude Code, GitHub Copilot, Windsurf, Codex, or your own custom agent. If it can run a shell command, it can query your test health data. ### Stop debugging noise When CI fails, your agent checks which tests are quarantined and skips them. It only investigates failures that are actually new. ### No plugins, no setup Pass `--json` and the CLI outputs structured data any tool can parse. Add it to your agent's instructions and it just works. > Integrating Mergify transformed our development process. It gives us full control over merges and schedules. It streamlined our workflow, helped catch issues early, and improved team efficiency and software reliability. Sean Davis Senior CI/CD Engineer at Ava Solutions ## Teams that merge more and break main less. [ ### 40% faster CI runs, fewer flaky failures Read story → AheadComputing 100 engineers · Semiconductor Before Mergify: GitHub's merge queue ](https://mergify.com/customers/aheadcomputing) [ ### Zero breaking changes on main since adopting Mergify Read story → Zama 30 engineers · Blockchain / Cryptography ](https://mergify.com/customers/zama) [ ### 2x throughput with no CI expansion Read story → Berkshire Grey 100 engineers · Robotics & Artificial Intelligence Before Mergify: Bulldozer ](https://mergify.com/customers/berkshiregrey) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## Your tests already run. Make them count. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/pricing # Every product. One plan.Zero compromises. Every plan includes CI Insights, Test Engine, and Merge Queue. Inactive contributors don't cost a thing. Not sure it's worth it? Calculate your team's ROI first. See what broken main and flaky tests cost your team each month, and how Mergify pays for itself. [See your ROI](https://mergify.com/calculator) Free / Open Source $0 Zero cost for open source projects and private teams of up to 5. [Get started](https://dashboard.mergify.com/) - ✓ Up to 5 users (on private repositories) - ✓ CI Insights - ✓ Test Engine - ✓ Merge Queue - ✗ Support via email - ✗ 24/7 Premium Support - ✗ Slack Connect Support Channel - ✗ Quarterly Roadmap Review - ✗ On-premise deployment Open source projects Max $21per seat / month\-15% Switch to yearly Prices in USD · Everything you need to keep CI fast and main green [Start a free 14-day trial](https://dashboard.mergify.com/) No credit card. One click from your dashboard. - ✓ Up to 100 users - ✓ CI Insights - ✓ Test Engine - ✓ Merge Queue - ✓ Support via email - ✗ 24/7 Premium Support - ✗ Slack Connect Support Channel - ✗ Quarterly Roadmap Review - ✗ On-premise deployment Teams on Max Enterprise Custom Dedicated support, custom retention, on-premise deployment, and quarterly roadmap reviews [Schedule a call](https://mergify.com/discovery) - ✓ Unlimited users - ✓ CI Insights - ✓ Test Engine - ✓ Merge Queue - ✓ Support via email - ✓ 24/7 Premium Support - ✓ Slack Connect Support Channel - ✓ Quarterly Roadmap Review - ✓ On-premise deployment (Enterprise Server) Enterprise customers What teams get after switching to Mergify [40%faster CI runsAheadComputing →](https://mergify.com/customers/aheadcomputing)[2×merge throughputCerebras →](https://mergify.com/customers/cerebras)[35%shorter PR-to-production loopProductboard →](https://mergify.com/customers/productboard) ## We count like engineers, not accountants A contributor is active if they open a PR or push commits to one in the last 30 days. You can see exactly who counts in your dashboard. [Learn how we calculate it](https://docs.mergify.com/billing/) > With Mergify, you can give power to the developers because it's in their own repo. They can extend and start using it more and more because we don't really pay based on executions; we pay based on seats. Back Market DevOps Engineering | | Open Source | Free | Max | Enterprise | | --- | --- | --- | --- | --- | | Users | Unlimited | Up to 5 users | Up to 100 users | Unlimited users | | Private Repositories | ✗ | ✓ | ✓ | ✓ | | CI Insights | | Job Metrics | ✓ | ✓ | ✓ | ✓ | | Data Retention | 24 hours | 24 hours | 30 days | Custom | | Job Auto-Retry | ✗ | ✓ | ✓ | ✓ | | Slack Notifications | ✗ | ✓ | ✓ | ✓ | | Test Engine | | Test Metrics | ✓ | ✓ | ✓ | ✓ | | Data Retention | 7 days | 7 days | 30 days | Custom | | Quarantine | ✗ | ✓ | ✓ | ✓ | | Tracked Branches | ✗ | ✗ | Up to 5 branches | Up to 5 branches | | Merge Queue | | Parallel Checks | ✓ | ✓ | ✓ | ✓ | | Batch | ✗ | ✓ | ✓ | ✓ | | Scopes | ✗ | ✓ | ✓ | ✓ | | Pause | ✗ | ✓ | ✓ | ✓ | | Pull Request Priorities | ✗ | ✓ | ✓ | ✓ | | Two-Steps CI | ✗ | ✓ | ✓ | ✓ | | Stacked Pull Requests | ✓ | ✓ | ✓ | ✓ | | Monitoring | ✗ | ✓ | ✓ | ✓ | | Pull Requests Dependencies | ✓ | ✓ | ✓ | ✓ | | Scheduled Freeze | ✗ | ✓ | ✓ | ✓ | | Scheduled Merge | ✓ | ✓ | ✓ | ✓ | | Security | | Data Encryption | ✓ | ✓ | ✓ | ✓ | | SSO | ✓ | ✓ | ✓ | ✓ | | Delegated Roles | ✗ | ✗ | ✗ | ✓ | | SOC | ✗ | ✗ | SOC 2 Type 2 | SOC 2 Type 2 | | On-premise deployment (Enterprise Server) | ✗ | ✗ | ✗ | Optional | | Support | | Email Support | ✗ | ✗ | ✓ | ✓ | | 24/7 Premium Support | ✗ | ✗ | ✗ | Optional | | Slack Connect Channel | ✗ | ✗ | ✗ | ✓ | | Quarterly Roadmap Review | ✗ | ✗ | ✗ | ✓ | | | [Get started](https://dashboard.mergify.com/) | [Get started](https://dashboard.mergify.com/) | [Start free trial](https://dashboard.mergify.com/) | [Contact us](https://mergify.com/discovery) | ## Startups move fast. So should your merges. We give startups up to $12,000 in credit to skip the CI chaos and focus on building. [Discover our Startup Program](https://mergify.com/startups) ## Built by paranoid engineers (the good kind) We're dead serious about protecting your workflows and your data. Ask us anything. [Check the locks and bolts](https://docs.mergify.com/security) ## Teams that merge more and break main less. [ ### 2x merge throughput with 3‑hour CI runs Read story → Cerebras 350 engineers · AI Infrastructure ](https://mergify.com/customers/cerebras) [ ### Cut CI time and kept main green Read story → Apex Fintech Solutions 350 engineers · Financial Technology ](https://mergify.com/customers/apex) [ ### 2x faster merges across a massive monorepo Read story → Back Market 300 engineers · E-commerce ](https://mergify.com/customers/back-market) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## Frequently asked questions How does seat-based billing work? You only pay for active contributors. A contributor is active if they open a pull request or push commits to one in the last 30 days. Inactive teammates, bots, and one-off contributors don't count, and you can see exactly who counts in your dashboard. What's the difference between monthly and yearly billing? Yearly billing on the Max plan saves 15% compared to paying monthly. You can switch between monthly and yearly at any time. Is there really a free plan? Yes. Mergify is free for open source projects and for private teams of up to 5 active contributors. You get all three products: Merge Queue, CI Insights, and Test Engine. The Open Source and Free plans don't cover the same features, so check the table above for what each one includes. What do I get on every plan? Every plan includes all three products: CI Insights, Test Engine, and Merge Queue. We don't gate whole products behind higher tiers. What changes between plans is seats, data retention, support, enterprise controls, and some advanced features inside each product. Can a large team evaluate Mergify without talking to sales? Yes. Start a 14-day trial of Max from your dashboard: one click, no credit card, and the full product running on your real repositories. You only need to talk to us if you want more than 100 seats, on-premise deployment, a signed MSA, or a company-wide agreement instead of per-seat billing, and none of that has to happen before you try it. When should I talk to sales about Enterprise? Enterprise fits when you need more than 100 seats, on-premise deployment, custom data retention, a signed MSA, 24/7 premium support, or quarterly roadmap reviews. On-premise means Mergify Enterprise Server: the full product running inside your own network, connected to GitHub Enterprise Server or GitHub.com. Pricing is custom and volume-based, so book a call and we'll size it with you. Do you offer discounts for startups? Yes. Our Startup Program gives eligible startups up to $12,000 in credit so you can skip the CI chaos while you're still finding product-market fit. --- Source: https://mergify.com/compare-mergify # How does Mergify compare? An honest, side-by-side look at the five merge queue platforms teams shortlist most often: GitHub's native queue, Graphite, Trunk, Aviator, and Mergify. We wrote this because the comparison content out there is either a competitor's marketing page or a year out of date. [Start a free 14-day trial](https://dashboard.mergify.com/)[Talk to our team](https://mergify.com/discovery) ## The short version [ ### GitHub Merge Queue Free, ships with GitHub. The right starting point for small teams. Best for: Teams under 10 engineers, single repo, simple CI. Where it falls short: No batching, no priorities, no monorepo scoping, no flaky test handling. Full comparison → ](https://mergify.com/compare/github-merge-queue)[ ### Graphite Stacked diffs and AI code review. Merge queue is included but narrower. Best for: Teams that want stacked PR workflows as the headline feature. Where it falls short: No CI observability, no flaky test detection at the queue layer, no merge governance. Full comparison → ](https://mergify.com/compare/graphite)[ ### Trunk Flaky test detection and test analytics at the CI layer. Best for: Teams whose primary pain is test stability and CI test visibility. Where it falls short: Narrow on merge governance and CI-wide observability outside test runs. Full comparison → ](https://mergify.com/compare/trunk)[ ### Aviator Capable merge queue at a lower starting price. Best for: Teams that only need queueing and want a low entry point. Where it falls short: Sells the queue alone. CI observability and merge governance are separate products. Full comparison → ](https://mergify.com/compare/aviator) ### Mergify Merge queue with batching plus bisect-on-failure, CI Insights (job health, auto-retry), Test Engine (flaky detection, quarantine, PR-level prevention), plus scheduled freezes and cross-repo PR dependencies in the queue. One plan, no separate products. Best for: Teams that have outgrown GitHub's queue, run monorepos, care about CI cost, or operate in regulated environments where tested SHA must equal merged SHA. Where it falls short: GitHub-only on the merge side. We ingest CI results from GitLab CI and most other CI providers, but the merge queue needs GitHub. Workflow Automation is being deprecated long-term, so do not buy it primarily for that. ## Feature matrix Side-by-side, no marketing fluff. Cells are check, dash, or short text where there is meaningful nuance. | Feature | GitHub MQ | Graphite | Trunk | Aviator | Mergify | | --- | --- | --- | --- | --- | --- | | CI batching (test multiple PRs together) | – | – | ✓ | ✓ | ✓ | | Bisect on batch failure | – | – | ✓ | ✓ | ✓ | | Speculative checks | ✓ | – | ✓ | ✓ | ✓ | | Multi-level priorities | Jump to front | – | Basic | Basic | Multiple levels | | Two-step CI (light pre-queue, heavy in-queue) | – | – | ✓ | ✓ | ✓ | | Monorepo / scope-aware batching | – | – | ✓ | ✓ | ✓ | | Flaky test detection | – | – | Separate product | – | ✓ | | Flaky test quarantine | – | – | Separate product | – | ✓ | | PR-level flaky test prevention | – | – | – | – | ✓ | | Auto-retry on CI failure | – | – | Anti-flake only | ✓ | Configurable rules | | Custom merge conditions (rule engine) | Status checks only | – | Status checks only | Basic | Full YAML | | Queue pause | – | – | ✓ | API only | UI + API | | Scheduled merge freezes | – | – | – | – | ✓ | | PR dependencies (Depends-On) | – | – | – | – | ✓ | | Stacked PRs | – | Core product | – | Separate product | Free, open source | | Merged SHA equals tested SHA | – | ✓ | ✓ | ✓ | ✓ | | CI Insights (job health, auto-retry analytics) | – | – | – | Basic analytics | Full product | Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama ## Which one should you pick? Match the tool to your actual pain. The wrong answer for your team is paying for features you do not need, or skipping features you do. ### Pick GitHub MQ if - ✓ You have fewer than 10 engineers contributing - ✓ You ship under 20 PRs per day on the queued repo - ✓ You do not yet have a flaky test problem - ✓ Your CI runs under 15 minutes per PR ### Pick a paid platform if - ✓ 20+ engineers ship daily, queue throughput matters - ✓ Your monorepo needs scope-aware test routing - ✓ Flaky tests are eating engineering time - ✓ You need merge freezes, PR dependencies, or scheduling - ✓ Audit posture requires tested SHA to equal merged SHA If your team is small and you just need basic queueing, GitHub Merge Queue is free and ships with GitHub. Among the paid options: if your pain is stacked PRs, look at Graphite. If you want the cheapest queue and only the queue, Aviator. If you want one platform that covers the queue, CI observability, and merge governance under one plan, Mergify. The teams we see consolidate fastest are the ones running two or three of these side by side and paying for overlap. That is the case Mergify is built to win. > Early on, as the team, tests, and CI complexity grew, it was quite clear that we needed an automated tool to resolve bottlenecks. Our team loves the way Mergify solved our growing pains. Tomasz Biernacki Quality Assurance Engineer at Pitch ## Related comparisons Other comparisons customers ask about that do not fit a merge queue matrix. [ ### Flaky test tools compared Mergify Test Engine, Trunk Flaky Tests, Datadog Test Optimization and BuildPulse: how each one detects flaky tests, and why Mergify handles them inside the job, on the pull request and in the merge queue. Read comparison → ](https://mergify.com/compare/flaky-test-tools)[ ### Mergify vs Datadog CI Visibility Datadog CI Visibility is a CI observability product, not a merge queue. Different category, but customers occasionally evaluate both. The comparison covers when each one fits. Read comparison → ](https://mergify.com/compare/datadog-ci-visibility) ## Frequently asked questions ### Which merge queue is the best for a small team? For teams of fewer than 10 engineers on a single repo, GitHub's native merge queue is the right starting point. It is free, ships with GitHub, and handles the basics. Once PR volume, monorepo complexity, or CI cost become problems, the other options become worth the budget. See [Mergify vs GitHub merge queue](https://mergify.com/compare/github-merge-queue). ### Mergify vs Graphite: how do they differ? Graphite leads with stacked diffs and AI code review, and their merge queue requires teams to adopt the Graphite platform end-to-end (not just the queue). Mergify covers the merge queue independently, plus CI Insights (job health, auto-retry), and Test Engine (flaky detection, quarantine, PR-level prevention) in every plan. The Mergify queue lands GitHub stacked pull requests in order, reading stack membership from GitHub itself rather than from branch chaining. See the [Mergify vs Graphite](https://mergify.com/compare/graphite) comparison. ### Mergify vs Trunk: how do they differ? Trunk's flaky test detection lives at the CI test layer. Mergify Test Engine integrates flaky handling with the merge queue itself, so the signal drives quarantine and smart auto-retry. On top of that, Mergify ships CI Insights for full job and runner observability (duration, failure rates, cost) that Trunk does not have. Mergify also covers merge governance (freezes, dependencies, scheduling) that Trunk does not. See the [Mergify vs Trunk](https://mergify.com/compare/trunk) comparison. ### Mergify vs Aviator: how do they differ? Aviator is a capable merge queue at a lower starting price, but it sells the merge queue alone, with CI observability and merge governance as separate products. Mergify includes all of those in every plan. See the [Mergify vs Aviator](https://mergify.com/compare/aviator) comparison. ### Can I use more than one merge queue on the same repo? Two different merge queue tools cannot own merging into the same branch (so you pick one tool per repo). Inside Mergify, though, parallel scope queues let you run multiple queues simultaneously on the same repo across independent parts of a monorepo, which is the common pattern for large monorepo deployments. ### What about Datadog CI Visibility? Datadog CI Visibility is a CI observability product, not a merge queue, so it is not on this comparison. Customers sometimes evaluate both, see the dedicated [Mergify vs Datadog CI Visibility](https://mergify.com/compare/datadog-ci-visibility) page for that comparison. ## Teams that merge more and break main less. [ ### 2x merge throughput with 3‑hour CI runs Read story → Cerebras 350 engineers · AI Infrastructure ](https://mergify.com/customers/cerebras) [ ### Cut CI time and kept main green Read story → Apex Fintech Solutions 350 engineers · Financial Technology ](https://mergify.com/customers/apex) [ ### 2x faster merges across a massive monorepo Read story → Back Market 300 engineers · E-commerce ](https://mergify.com/customers/back-market) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## Move faster. Break less. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Talk to our team](https://mergify.com/discovery) --- Source: https://mergify.com/compare/aviator # Mergify vs Aviator Aviator sells its merge queue next to a review inbox, AI code verification, and release tooling. Mergify is built around the path from approval to main, with CI Insights and Test Engine in the same plan as the queue. [Start a free 14-day trial](https://dashboard.mergify.com/)[See pricing](https://mergify.com/pricing) ## Feature comparison | Feature | Aviator | Mergify | | --- | --- | --- | | Speculative checks | ✓ | ✓ | | CI batching with bisect-on-failure | ✓ | ✓ | | Merge when a later batch proves an earlier failure was flaky | ✓ | ✓ | | Two-step CI | ✓ | ✓ | | Monorepo lanes from your build graph (Bazel, Nx, Turborepo, Pants) | Scale plan ($50/dev) | Free and Max plans | | PR priorities | ✓ | Multiple levels | | Queue pause | UI + API | UI + API | | Scheduled freeze with exceptions (e.g. hotfixes) | Cron pause/unpause | ✓ | | Flaky test detection + quarantine | Queue-level handling | Full product (Test Engine) | | Custom merge conditions (rule engine) | Fixed list of options | Full YAML rule engine | | Queue analytics | ✓ | ✓ | | Monitoring integrations (Datadog) | ✓ | ✓ | | Slack notifications | ✓ | ✓ | | CI Insights (job health, auto-retry) | – | Full product | | Queue event log | Per-PR timeline | Queue-wide | | AI code verification | Verify (early access) | – | | Billing by active contributors only | Per developer | Active contributors | | Fast-forward merge | ✓ | ✓ | | On-premise deployment | Enterprise | Enterprise | [Get started](https://dashboard.mergify.com/) Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama ## More than a merge queue On Aviator, the queue starts on the Team plan and monorepo lanes on Scale. Mergify includes the queue, CI observability, test intelligence, and merge governance in every plan. ✓ Three products in one plan: Merge Queue, CI Insights, and Test Engine ✓ Monorepo scopes on the Free and Max plans, not a higher tier ✓ Scheduled freezes with exceptions, so a hotfix can still land during a release window ✓ Full YAML rule engine for merge conditions on labels, files, authors, and schedules ✓ Fair billing: only active contributors count (inactive users are free) ✓ Running since 2018 with 8 years of GitHub edge case handling in production ## Evaluating merge queue tools? We'll walk you through how Mergify compares for your specific setup. No pitch, just answers. [Book a discovery call](https://mergify.com/discovery) > Early on, as the team, tests, and CI complexity grew, it was quite clear that we needed an automated tool to resolve bottlenecks. Our team loves the way Mergify solved our growing pains. Tomasz Biernacki Quality Assurance Engineer at Pitch ## What Aviator doesn't cover Aviator's queue covers the basics well. These are the differences teams run into as the repository grows. ### Monorepo lanes cost extra Affected targets, Aviator's monorepo-aware queue, is on the Scale plan at $50 per developer per month. Mergify's scopes are on the Free and Max plans. ### A failure resets the queue In Aviator's parallel mode, a failing draft PR is removed and every draft behind it is closed and re-validated. In Mergify's parallel mode, only batches that share a scope are chained, so a failure in one lane leaves the others running. ### Freezes are cron rules Aviator can pause and unpause the queue on a cron schedule. Mergify's scheduled freezes take exceptions, so a pull request labelled hotfix can still merge while everything else waits. ### Merge conditions from a fixed list Aviator's pre-queue conditions cover approvals, required checks, resolved conversations, and a regex on the title and body. Mergify's rule engine combines labels, files, authors, and time windows however you need. ### No test-level flaky detection Aviator handles flaky failures inside the queue. Knowing which test is flaky, how often, and quarantining it is Test Engine, which every Mergify plan includes. ### Audit log on Enterprise only Aviator's audit log is an Enterprise feature and records configuration changes. Queue activity lives on each PR's timeline. Mergify keeps a queue-wide event log. > Mergify is a no-brainer. If you want to have your main branch always working then you have to introduce some merge queue functionality. Paco Sevilla DevOps Engineer at DeepDrive ## The pricing question Aviator's merge queue starts on its Team plan at $20 per developer per month, and the monorepo-aware queue needs Scale at $50. Mergify Max is $21 per active contributor per month, with scopes, CI Insights, and Test Engine included. Mergify is free for up to 5 active contributors on private repos, merge queue included. Aviator's free plan does not include the merge queue. [See Mergify pricing](https://mergify.com/pricing)[Book a discovery call](https://mergify.com/discovery) ## Teams that merge more and break main less. [ ### 2x merge throughput with 3‑hour CI runs Read story → Cerebras 350 engineers · AI Infrastructure ](https://mergify.com/customers/cerebras) [ ### Cut CI time and kept main green Read story → Apex Fintech Solutions 350 engineers · Financial Technology ](https://mergify.com/customers/apex) [ ### 2x faster merges across a massive monorepo Read story → Back Market 300 engineers · E-commerce ](https://mergify.com/customers/back-market) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) Last updated: September 29, 2026 ## Try the complete CI platform Merge queue, CI Insights, and Test Engine. All included, up to 5 users free. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/compare/buildpulse # A BuildPulse alternative Mergify Test Engine retries known flaky tests inside the job, quarantines them without code in your test suite and lets them out once they recover. It runs on any CI, with a free plan and no test-volume metering. Last updated: October 28, 2026 [Start a free 14-day trial](https://dashboard.mergify.com/)[See Test Engine](https://mergify.com/product/test-engine) ## The short answer Mergify Test Engine works from inside your test framework, on any CI, and ships with the merge queue. - ✓ Flaky failures handled inside the job. BuildPulse reads the reports your CI uploads, so a flaky test has already failed the build when it finds out. Test Engine retries a known flaky test right where it failed. - ✓ Quarantine you do not have to wire up. BuildPulse serves its quarantine list through an API, and skipping those tests is left to your test suite. Test Engine applies quarantine itself, on any branch. - ✓ Tests come back on their own. A BuildPulse quarantine ends when its ticket is closed. Test Engine can release a test once it is healthy again on the default branch. - ✓ New flaky tests stopped at the pull request. Prevention reruns the tests a pull request adds or changes and blocks the merge if one is flaky. - ✓ Priced per seat, with a free plan. BuildPulse has a 14-day trial and plans sized by tests run each month. Mergify is free up to 5 contributors, and the seat price includes Merge Queue and CI Insights. ## BuildPulse vs Mergify Test Engine From each vendor's own documentation and pricing page. Sources are listed at the bottom of the page. BuildPulse compared with Mergify Test Engine | Feature | BuildPulse | Mergify Test Engine | | --- | --- | --- | | How flaky tests are found | Analyzes results across CI builds. Tests that breach your thresholds are flagged and can be auto-quarantined | Passed and failed on the same commit. Every test is healthy, flaky or broken, so a real regression is never filed as noise | | Retries inside the job | No. BuildPulse works from the XML reports your CI uploads after the job | Yes. A known flaky test is retried right where it failed, inside your test framework, so the job goes green without waiting for quarantine | | New flaky tests on the pull request | Not documented | Prevention reruns the tests a pull request adds or changes, on the same commit, and blocks the merge if one is flaky | | Quarantine | Automatic or manual. Quarantined tests are served by the Quarantine API, and your test suite has to skip them itself | Automatic once the evidence holds, so one bad infra run does not quarantine half your suite. Quarantined tests keep running on any branch, and their failures stop blocking merges | | Leaving quarantine | When the linked ticket is moved to Done, or by hand | Automatically, once the test is healthy again on the default branch, or by hand | | Merge queue | None | Mergify Merge Queue, same plan. Queue runs get the same retries and quarantine, and Fail First bisection isolates a real failure fast | | CI providers | 8 named, with GitLab CI, Jenkins and Buildkite in beta. Azure DevOps, AWS and TeamCity through extra configuration | Any CI. Native GitHub Actions, Buildkite and Jenkins integrations, everything else through the Mergify CLI | | Test frameworks | XML test reports, with guides for 8 frameworks: Cypress, Go, Jest, Minitest, Mocha, PHPUnit, pytest and RSpec | Native in your test framework: pytest, Jest, RSpec, Vitest, Playwright, Go, PHPUnit, JUnit and more (15 frameworks documented) | | Ticketing and alerts | Jira, with an issue opened when a test is quarantined. Slack reports and alerts | Jira and Linear coming. Slack notifications and an event log today | | AI agents | MCP server with flaky test history. AI code review in invite-only beta | The Mergify CLI gives your coding agent each test's health and quarantine state, so it fixes the right tests | | Pricing | By tests run per month: $99 for 3M, $249 for 10M, $499 for 30M, Enterprise custom. 14-day trial, no free plan. Included with BuildPulse Runners | $21 per active seat a month, with Merge Queue and CI Insights in the same plan. No test-volume metering. Free up to 5 contributors | ## Why teams look for an alternative BuildPulse finds flaky tests across many CI providers, and since its relaunch it also sells CI runners, code coverage and engineering metrics. On flaky tests, Test Engine goes further in four places. The first is timing. BuildPulse learns about a test from the report your CI uploads once the job is over, so the first flaky failures still land on a developer before anything can be quarantined. Test Engine runs inside your test framework and retries a known flaky test while the job is still running. The second is the wiring. BuildPulse quarantines by serving a list through its API, and its docs send you to your test framework's documentation to skip those tests. That is code to write and keep working in every repository. Test Engine applies quarantine itself, keeps quarantined tests running so their history stays current, and can release them once they are healthy again, without waiting for someone to close a ticket. The third is the merge queue. BuildPulse has none, so a flaky test that slips through still throws pull requests out of whatever queue you run. Test Engine's retries and quarantine apply to queue runs, next to [Mergify Merge Queue](https://mergify.com/product/merge-queue) in the same plan. And the price grows with your test suite. BuildPulse plans are sized by tests run each month, from $99 for 3 million. Mergify charges per seat for Merge Queue, [CI Insights](https://mergify.com/product/ci-insights) and [Test Engine](https://mergify.com/product/test-engine), with no metering on test volume and a free plan up to 5 contributors. ## Moving from BuildPulse 1. Install the Mergify GitHub App and turn on Test Engine. It is free for up to 5 contributors, then on the [standard seat price](https://mergify.com/pricing). 2. Point the Mergify CLI at the XML reports your jobs already write for BuildPulse, or add the Test Engine plugin for your framework. 3. Remove the step that reads BuildPulse's Quarantine API and skips tests. Test Engine handles quarantine on its own. 4. If you run BuildPulse Runners for compute, keep them. Test Engine works on any runner. Comparing more than one tool? See [Mergify, Trunk, Datadog and BuildPulse side by side](https://mergify.com/compare/flaky-test-tools). ## Frequently asked questions ### What is the best alternative to BuildPulse? Mergify Test Engine. It retries known flaky tests inside the job, blocks new flaky tests on the pull request, quarantines automatically without code in your test suite and releases a test once it is healthy again. It works on any CI, with a free plan and Merge Queue and CI Insights in the same seat price. ### How does BuildPulse quarantine flaky tests? Tests that breach your thresholds are marked as quarantined and served through BuildPulse's Quarantine API. Your test suite has to read that list and skip those tests, following your framework's own docs. A test leaves quarantine when its ticket is moved to Done, or by hand. Mergify applies quarantine itself and can release a test automatically when it recovers. ### Does BuildPulse retry flaky tests? No. BuildPulse works from the XML reports your CI uploads after the job, so it acts through quarantine. Mergify retries a known flaky test inside the job, so the build goes green on the first run. ### Does BuildPulse have a free plan? No. Each plan has a 14-day free trial, then starts at $99 a month for 3 million tests. Flaky test detection is also included with BuildPulse Runners. Mergify is free for up to 5 contributors, then $21 per active seat a month with no test-volume metering. [See Mergify pricing](https://mergify.com/pricing). ### Can I keep BuildPulse Runners and use Mergify for flaky tests? Yes. Test Engine reads test results from whatever runner executes your jobs, so BuildPulse Runners, GitHub-hosted runners and self-hosted runners all work. ## Sources Every claim above comes from the vendor's own documentation or pricing page, read on 2026-10-28. - [Mergify: Test Engine](https://docs.mergify.com/test-engine) - [Mergify: Prevention](https://docs.mergify.com/test-engine/prevention) - [Mergify: Mitigation](https://docs.mergify.com/test-engine/mitigation) - [Mergify: Quarantine](https://docs.mergify.com/test-engine/quarantine) - [BuildPulse: Home](https://buildpulse.io) - [BuildPulse: Test quarantining](https://docs.buildpulse.io/flaky-tests/guides/Test%20Quarantining) - [BuildPulse: CI providers](https://docs.buildpulse.io/Guides/CI%20Providers/Intro) - [BuildPulse: Integrations](https://buildpulse.io/integrations) - [BuildPulse: Pricing](https://buildpulse.io/pricing) ## Flaky tests, handled inside the job. Test Engine, CI Insights and Merge Queue in one plan. Free for up to 5 contributors. [Get started](https://dashboard.mergify.com/)[See how Test Engine works](https://mergify.com/product/test-engine) --- Source: https://mergify.com/compare/datadog-ci-visibility # CI Visibility That Takes Action Datadog can find and quarantine a flaky test. Mergify does that too, then carries the same signal into what gets merged. [Start a free 14-day trial](https://dashboard.mergify.com/)[See pricing](https://mergify.com/pricing) ## Feature comparison | Feature | Datadog | Mergify | | --- | --- | --- | | Flaky test detection | ✓ | ✓ | | Test quarantine | Quarantine or disable | ✓ | | Auto-retry on failure | Test-level, in the tracer | Job-level, configurable rules | | Pipeline traces | ✓ | Test-focused analytics | | CI framework support | Broad (APM-based) | 15+ test frameworks | | Merge queue integration | – | Native | | GitHub-native workflow | Separate platform | ✓ | | Scheduled freezes and PR dependencies | – | Included | | Setup complexity | Agent + library instrumentation | JUnit upload or GitHub App | [Get started](https://dashboard.mergify.com/) Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama ## From test health to merge decisions Datadog's flaky test tooling ends at the test run. Mergify's keeps going until the pull request is on main. ✓ Flaky tests quarantined automatically. No more blocked pipelines. ✓ CI-native. Built for GitHub Actions and Jenkins, not bolted onto APM. ✓ Action rules with no human in the loop: retry, skip, or quarantine on a condition. ✓ Complete merge platform. CI insights feed merge queue decisions. ✓ No Datadog dependency. Standalone CI intelligence, no existing spend required. ✓ Predictable pricing. No per-committer add-on billing surprises. > Before CI Insights, we were wasting hours debugging CI issues. We now cut triage time in half within a week. Liam Hartman Platform Engineer at Vertex ## CI Insights is just the beginning Mergify connects test intelligence directly to your merge workflow. Detect problems, act on them automatically. ### Detect Flaky tests, slow jobs, failure patterns ### Act Quarantine, auto-retry, skip known failures ### Merge Smart queue decisions using CI health data ## Evaluating CI visibility tools? We'll walk you through how Mergify compares for your specific setup. No pitch, just answers. [Book a discovery call](https://mergify.com/discovery) ## What Datadog CI Visibility doesn't cover Datadog gives you visibility. These are the gaps teams notice when they need action. ### Quarantine stops at the test A quarantined test no longer fails the build, which is the point. Whether the pull request is then safe to merge against today's main is a question Datadog leaves to someone else. ### Retries live inside the test run Auto Test Retries re-run a failing test case through the language library. Re-running a whole CI job that died on a runner or a network timeout is not part of it. ### No merge queue Datadog observes your CI but doesn't touch your merge process. PRs still pile up and conflict. ### Requires Datadog platform CI Visibility is an add-on to the broader Datadog platform. You need an existing Datadog contract to use it. ### Per-committer pricing CI Visibility bills per committer on top of your existing Datadog spend. Costs grow with your team, fast. > We had 40 known-flaky tests and no system to deal with them. Engineers just re-ran the suite and hoped. Test Engine quarantined the worst offenders on day one, and within a month we actually fixed most of them because we could finally see which ones mattered. Lucas Meier Staff Software Engineer at Norde ## The pricing question Datadog CI Visibility is a per-committer add-on to the broader Datadog platform. You need an existing Datadog contract, then pay extra for CI visibility on top. Costs scale with every developer who pushes code. Mergify includes CI Insights in every plan. Predictable per-seat pricing, no surprise line items. Free for up to 5 users on private repos. [See Mergify pricing](https://mergify.com/pricing)[Book a discovery call](https://mergify.com/discovery) ## Teams that merge more and break main less. [ ### 2x merge throughput with 3‑hour CI runs Read story → Cerebras 350 engineers · AI Infrastructure ](https://mergify.com/customers/cerebras) [ ### Cut CI time and kept main green Read story → Apex Fintech Solutions 350 engineers · Financial Technology ](https://mergify.com/customers/apex) [ ### 2x faster merges across a massive monorepo Read story → Back Market 300 engineers · E-commerce ](https://mergify.com/customers/back-market) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## Your CI pipeline should fix itself. CI Insights, test quarantine, and merge queue. All included, up to 5 users free. [Get started](https://dashboard.mergify.com/)[See how CI Insights works](https://mergify.com/product/ci-insights) --- Source: https://mergify.com/compare/datadog-test-optimization # A Datadog Test Optimization alternative Mergify Test Engine retries known flaky tests inside the job, stops new ones at the pull request and keeps them from throwing pull requests out of your merge queue. It runs on any CI, with no Datadog contract. Last updated: October 28, 2026 [Start a free 14-day trial](https://dashboard.mergify.com/)[See Test Engine](https://mergify.com/product/test-engine) ## The short answer Mergify Test Engine works from inside your test framework, on any CI, and ships with the merge queue. - ✓ It retries the tests that are actually flaky. Datadog's Auto Test Retries reruns any failing test up to 5 times by default, so a real bug that passes once goes green. Test Engine retries a test it already knows is flaky, inside your test framework. - ✓ The pull request gate works with a merge queue. Datadog's own docs say PR Gates cannot be set as required on GitHub repositories that use a merge queue. Prevention blocks a flaky new test from merging either way. - ✓ Quarantine for every framework. Datadog's flaky test management needs its native instrumentation. Test Engine quarantines tests from any framework, JUnit XML uploads included. - ✓ The merge queue is in the same plan. Queue runs get the same retries and quarantine, so a flaky test never throws a pull request out of the queue. - ✓ No Datadog contract and no span metering. One seat price covers Merge Queue, CI Insights and Test Engine, with nothing metered on test volume. ## Datadog Test Optimization vs Mergify Test Engine From each vendor's own documentation and pricing page. Sources are listed at the bottom of the page. Datadog Test Optimization compared with Mergify Test Engine | Feature | Datadog Test Optimization | Mergify Test Engine | | --- | --- | --- | | What counts as flaky | A test that passed and failed on the same commit | Passed and failed on the same commit. Every test is healthy, flaky or broken, so a real regression is never filed as noise | | Retries inside the job | Auto Test Retries reruns any failing test, up to 5 times by default. Restricting retries to known flaky tests is documented for Java only | A known flaky test is retried right where it failed, inside your test framework, so the job goes green without retrying real failures | | New flaky tests on the pull request | Early Flake Detection retries new tests up to 10 times. A PR Gate can block the merge, but it cannot be required on a GitHub repository that uses a merge queue | Prevention reruns the tests a pull request adds or changes, on the same commit, and blocks the merge if one is flaky, with or without a merge queue | | Quarantine | Quarantine or disable a test, by hand or through Flaky Test Policies. Requires Datadog's native instrumentation for your test framework | Automatic once the evidence holds, so one bad infra run does not quarantine half your suite. Released automatically when the test is healthy again. Works from JUnit XML too | | Test frameworks | Native libraries for .NET, Java, JavaScript, Python, Ruby, Swift and Go. Anything else uploads JUnit XML, which gets fewer features | Native in your test framework: pytest, Jest, RSpec, Vitest, Playwright, Go, PHPUnit, Rust, JUnit and more (15 frameworks documented) | | Merge queue | None | Mergify Merge Queue, same plan. Queue runs get the same retries and quarantine, and Fail First bisection isolates a real failure fast | | CI providers | Any CI provider | Any CI. Native GitHub Actions, Buildkite and Jenkins integrations, everything else through the Mergify CLI | | Ticketing | Jira issues and Datadog work items | Jira and Linear coming. Slack notifications and an event log today | | AI agents | Bits Code generates a fix and can open a pull request (preview) | The Mergify CLI gives your coding agent each test's health and quarantine state, so it fixes the right tests | | Pricing | $20 per committer a month billed yearly, $29 on demand. 1M test spans per committer, then $3 per extra million | $21 per active seat a month, with Merge Queue and CI Insights in the same plan. No test-volume metering. Free up to 5 contributors | ## Why teams look for an alternative Datadog Test Optimization makes sense when the rest of your observability already lives in Datadog. On flaky tests, Test Engine goes further in four places. The first is retries. By default Datadog's Auto Test Retries reruns any failing test up to 5 times, so a test that fails because of a real race condition in your code can pass on the third try and turn the build green. Test Engine retries only the tests it already knows are flaky, inside your test framework, and lets a real failure fail. The second is the merge queue. Early Flake Detection and a "new flaky tests" PR Gate can catch a flaky test before it merges, but Datadog documents that a PR Gate cannot be required on a GitHub repository that uses a merge queue. A gate nobody is required to pass is a suggestion. Test Engine's Prevention blocks the merge on its own, and its retries and quarantine apply to the queue's CI runs as well, next to [Mergify Merge Queue](https://mergify.com/product/merge-queue) in the same plan. The third is coverage. Quarantine and disable need Datadog's native instrumentation, which exists for seven languages. A PHPUnit or Rust suite can upload JUnit XML for visibility, but its flaky tests keep breaking builds. Test Engine quarantines from the same JUnit XML. And the bill comes on top of the Datadog contract, per committer, with test spans metered past 1M per committer. Mergify charges one seat price for Merge Queue, [CI Insights](https://mergify.com/product/ci-insights) and [Test Engine](https://mergify.com/product/test-engine). ## Moving from Datadog Test Optimization 1. Install the Mergify GitHub App and turn on Test Engine. It is free for up to 5 contributors, then on the [standard seat price](https://mergify.com/pricing). 2. Add the Test Engine plugin for your framework, or point the Mergify CLI at the JUnit XML your jobs already write. Nothing else in your CI changes. 3. Turn off Datadog's Auto Test Retries once Test Engine has history on your suite, so a real failure is no longer retried away. 4. Keep Datadog for APM, logs and infrastructure. Test Engine only replaces the test layer. Comparing more than one tool? See [Mergify, Trunk, Datadog and BuildPulse side by side](https://mergify.com/compare/flaky-test-tools). ## Frequently asked questions ### What is the best alternative to Datadog Test Optimization? Mergify Test Engine. It retries known flaky tests inside the job, blocks new flaky tests on the pull request, quarantines automatically on any framework and releases a test once it is healthy again. It works on any CI, and Merge Queue and CI Insights come in the same seat price. ### Can Datadog PR Gates block merges when I use a merge queue? Not as a required check. Datadog's PR Gates documentation states: "PR Gates can not be set as required on GitHub repositories using a merge queue." Mergify's Prevention blocks a flaky new test from merging whether or not you use a merge queue. ### Does Datadog retry flaky tests automatically? Yes. Auto Test Retries reruns any failing test case up to 5 times by default, whether or not it is known to be flaky. Datadog documents an option to retry only known flaky tests for Java. Mergify retries only tests it has already classified as flaky. ### Do I need the Datadog tracer to quarantine flaky tests? Yes. Datadog states that Flaky Test Management features require its native instrumentation for your test framework. JUnit XML uploads through datadog-ci give you test visibility without quarantine or disable. Mergify quarantines tests whether they come from a native plugin or from JUnit XML. ### How much does Datadog Test Optimization cost? From $20 per committer a month billed yearly, or $29 on demand. Each committer includes 1M test spans a month, and extra spans cost $3 per million. A committer is anyone with at least three commits in the month. Mergify is $21 per active seat a month for Merge Queue, CI Insights and Test Engine together, with no test-volume metering, and free for up to 5 contributors. [See Mergify pricing](https://mergify.com/pricing). ### Can I keep Datadog and use Mergify for tests? Yes. Test Engine only replaces the test layer and reads your test results from CI, so Datadog can stay in place for APM, logs and infrastructure monitoring. ### Is this the same as Datadog CI Visibility? No. CI Visibility covers pipelines and jobs, and Test Optimization covers individual tests. For the pipeline side, see [Mergify vs Datadog CI Visibility](https://mergify.com/compare/datadog-ci-visibility). ## Sources Every claim above comes from the vendor's own documentation or pricing page, read on 2026-10-28. - [Mergify: Test Engine](https://docs.mergify.com/test-engine) - [Mergify: Prevention](https://docs.mergify.com/test-engine/prevention) - [Mergify: Mitigation](https://docs.mergify.com/test-engine/mitigation) - [Mergify: Quarantine](https://docs.mergify.com/test-engine/quarantine) - [Mergify: Detection](https://docs.mergify.com/test-engine/detection) - [Datadog: Auto Test Retries](https://docs.datadoghq.com/tests/flaky_tests/auto_test_retries/) - [Datadog: Early Flake Detection](https://docs.datadoghq.com/tests/flaky_tests/early_flake_detection/) - [Datadog: Flaky Test Management](https://docs.datadoghq.com/tests/flaky_management/) - [Datadog: PR Gates](https://docs.datadoghq.com/pr_gates/) - [Datadog: Test Optimization setup](https://docs.datadoghq.com/tests/setup/) - [Datadog: JUnit XML upload](https://docs.datadoghq.com/tests/setup/junit_xml/) - [Datadog: Pricing](https://www.datadoghq.com/pricing/?product=test-optimization) ## Flaky tests, handled inside the job. Test Engine, CI Insights and Merge Queue in one plan. Free for up to 5 contributors. [Get started](https://dashboard.mergify.com/)[See how Test Engine works](https://mergify.com/product/test-engine) --- Source: https://mergify.com/compare/depot vs # Mergify vs Depot Depot runs your CI jobs. Mergify decides which pull requests land on main. Here is where each one fits, and how teams run both. [Start a free 14-day trial](https://dashboard.mergify.com/)[See pricing](https://mergify.com/pricing) ## The short answer Depot sells compute: faster GitHub Actions runners, a GitHub Actions-compatible CI engine and container builds. Mergify sells no compute. Its merge queue and Test Engine work with whatever CI you use, and CI Insights reads the GitHub Actions jobs that run on Depot's runners. They are different layers of the same path, so they work together:**Depot makes each check faster, Mergify decides what is safe to merge.** ## Feature comparison | Feature | Depot | Mergify | | --- | --- | --- | | Hosted GitHub Actions runners | Core product | – | | GitHub Actions-compatible CI engine | Depot CI | – | | Container builds and build cache | ✓ | – | | Merge queue | – | Mature, since 2018 | | Batching with bisect on failure | – | ✓ | | Two-step CI | – | ✓ | | Monorepo / scope-aware queue | – | ✓ | | Flaky test signal | On Depot jobs | Any CI | | Flaky test quarantine | – | Test Engine | | New flaky tests caught on the PR | – | Test Engine | | Scheduled freezes and PR dependencies | – | Included | | Pricing model | Plan + compute usage | Active contributors only | [Get started](https://dashboard.mergify.com/) Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama ## A faster runner does not decide what lands > "The bottleneck has shifted from writing code to everything after." [depot.dev homepage](https://depot.dev/) > "we can skip CI entirely after the code is committed." [Depot, CI is changing](https://depot.dev/blog/ci-is-changing) We agree with the diagnosis. Coding agents and bigger teams open more pull requests than the path to main was built for, and the wait moved from writing code to everything after it. Faster compute shortens each run. It does not catch two pull requests that pass on their own and break main together, because each was tested against a main that no longer exists by the time it merges. That is the job of a merge queue: test every pull request, or a batch of them, against the main it will actually land on, and keep the ones that fail out. Skipping CI after the commit, as Depot's essay suggests, moves that risk onto main rather than removing it. Mergify merges more than half a million pull requests a month this way. Run it on Depot and you get both halves: every check the queue runs finishes sooner, and nothing lands that has not passed on top of what merged before it. ## What each tool does best One runs the jobs, the other decides what the results mean for main. ### Depot is great for - ✓ Faster GitHub Actions runners, switched with one runs-on label - ✓ A GitHub Actions-compatible CI engine you can migrate workflows into - ✓ Fast container builds with a shared cache - ✓ Sandboxes for agents that need to run code ### Mergify is great for - ✓ Testing each pull request against the main it will actually land on - ✓ Batching with bisect on failure when PR volume climbs - ✓ Scope-aware lanes for monorepos - ✓ Flaky test detection and quarantine on whatever CI runs the tests - ✓ Merge freezes and PR dependencies across repositories - ✓ Landing GitHub stacked pull requests in order ## Teams that merge more and break main less. [ ### 2x merge throughput with 3‑hour CI runs Read story → Cerebras 350 engineers · AI Infrastructure ](https://mergify.com/customers/cerebras) [ ### Cut CI time and kept main green Read story → Apex Fintech Solutions 350 engineers · Financial Technology ](https://mergify.com/customers/apex) [ ### 2x faster merges across a massive monorepo Read story → Back Market 300 engineers · E-commerce ](https://mergify.com/customers/back-market) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## FAQ ### Do I need a merge queue if I use Depot CI? If more than a handful of people merge to the same branch, yes. Depot makes each CI run faster. A merge queue answers a different question: whether a pull request that passed on its own branch still passes on top of everything that merged since. Two green pull requests can break main together, however fast the runner that tested them. Faster CI does make the queue cheaper to run, since every batch it tests finishes sooner. ### Can I run Mergify's merge queue on Depot CI or Depot runners? Yes. Mergify tests each queued pull request or batch in a draft pull request, so any workflow that triggers on `pull_request` runs for it. Depot CI lists `pull_request` and GitHub's `merge_group` trigger as supported on its [compatibility page](https://depot.dev/docs/ci/compatibility). With Depot's GitHub Actions runners nothing changes at all: your workflows stay in GitHub Actions and only the `runs-on` label moves. ### Does Depot detect flaky tests? Depot's test results include a "Possibly flaky tests" card that, in Depot's words, "surfaces tests that have both passed and failed on the same commit". Its test commands "only work inside Depot CI and Depot GitHub Actions runner jobs", and the docs describe no quarantine. Mergify Test Engine reads JUnit results from any CI, quarantines known flaky tests so they stop blocking merges, and flags new flaky tests on the pull request that introduces them. ### Is Mergify a CI provider? No. Mergify does not run your jobs or sell compute minutes. It sits between an approved pull request and main: the merge queue, CI Insights on the jobs your CI already runs, and Test Engine for flaky tests. Keep the CI you have, Depot included. ### How does pricing compare? They bill for different things, so teams running both pay for both. Depot charges a plan plus compute usage. Mergify charges per active contributor: anyone who opens a pull request or pushes to one in a 30-day sliding window. Every plan includes all three products, and the free plan covers up to 5 users. [See pricing](https://mergify.com/pricing). Last updated: October 30, 2026 ## Keep your CI. Put a queue in front of main. All three products included, up to 5 users. No credit card required. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/compare/flaky-test-tools # Flaky Test Tools Compared Mergify Test Engine, Trunk Flaky Tests, Datadog Test Optimization and BuildPulse all detect flaky tests. What sets them apart is what happens next: whether the failure is handled inside the job, before the merge, and inside your merge queue. Last updated: October 21, 2026 [Start a free 14-day trial](https://dashboard.mergify.com/)[See Test Engine](https://mergify.com/product/test-engine) ## The short answer [Mergify Test Engine](#mergify) detects and fixes flaky failures from inside your test framework, on any CI, and it ships with the merge queue. - ✓ Retried inside the job. A known flaky test is retried right where it failed, so the build goes green without anyone clicking rerun. Trunk can only quarantine the test after it has already failed your build. - ✓ Blocked on the pull request. Prevention reruns the tests a pull request adds or changes and stops the merge if one of them is flaky. - ✓ Quarantine that runs itself. A test goes into quarantine once the evidence holds and comes out when it is healthy again. - ✓ Built for the merge queue. Queue runs get the same retries and quarantine, so a flaky test never dequeues a pull request, and Fail First bisection finds the culprit fast when a batch really breaks. - ✓ One seat price. Merge Queue, CI Insights and Test Engine together, with no metering on test volume. ## Feature comparison From each vendor's own documentation and pricing page. Sources are listed at the bottom of the page. Flaky test tools compared: Trunk Flaky Tests, Datadog Test Optimization, BuildPulse and Mergify Test Engine | Feature | Trunk Flaky Tests | Datadog Test Optimization | BuildPulse | Mergify Test Engine | | --- | --- | --- | --- | --- | | What counts as flaky | Monitors you configure: pass-on-retry on the same commit, failure rate, failure count | Passed and failed on the same commit | Passed and failed on the same code, plus a failure-rate threshold (30% by default) | Passed and failed on the same commit. Every test is healthy, flaky or broken, so a real regression is never filed as noise | | Retries a flaky test inside the job | No. Trunk works from the results you upload and acts through quarantine | Auto Test Retries reruns any failing test up to 5 times by default, real failures included | No | Yes. A known flaky test is retried right where it failed, inside your test framework, so the job goes green without waiting for quarantine | | Catches a new flaky test on the pull request | No. Trunk reads the results you upload. Its new-test monitor only labels a test it has not seen before | Early Flake Detection retries new tests up to 10 times. A PR Gate can block the merge | Not documented | Prevention reruns the tests a pull request adds or changes, on the same commit, and blocks the merge if one is flaky | | Quarantine | Automatic for tests already flagged flaky, or manual. The Trunk CLI rewrites the job's exit code when only quarantined tests failed | Manual, or by policy on time, branch or failure rate | Automatic or manual. Your pipeline reads the Quarantine API and skips those tests itself | Automatic once the evidence holds, so one bad infra run does not quarantine half your suite. Released automatically when the test is healthy again. Quarantined tests keep running on any branch | | Merge queue | Trunk Merge Queue, included in the same plans | None. PR Gates cannot be required on GitHub repos that use a merge queue | None | Mergify Merge Queue, same plan. Queue runs get the same retries and quarantine, and Fail First bisection isolates a real failure fast | | CI providers | 14 named, including GitHub Actions, GitLab, Jenkins, CircleCI and Buildkite, plus others | Any CI provider | 8 named, including GitHub Actions, CircleCI, GitLab CI and Jenkins (both beta), plus others | Any CI. Native GitHub Actions, Buildkite and Jenkins integrations, everything else through the Mergify CLI | | How results get in | JUnit XML upload through the Trunk CLI, with guides for 28 frameworks | Datadog libraries for .NET, Java, JavaScript, Python, Ruby, Swift and Go, or JUnit XML upload | JUnit XML upload, with guides for 8 frameworks | Native in your test framework: pytest, Jest, RSpec, Vitest, Playwright, Go, JUnit and more (15 frameworks documented) | | Ticketing | Jira, Jira Data Center, Linear and Asana, plus webhooks | Jira issues and Datadog work items | Jira, Linear, GitHub Projects (beta) | Jira and Linear coming. Slack notifications and an event log today | | AI agents | Beta behind a waitlist. Trunk investigates, a Cursor Automation opens the PR | Bits Code generates a fix and can open a pull request | None. Owners get failure patterns, frequency data and linked commits | The Mergify CLI gives your coding agent each test's health and quarantine state, so it fixes the right tests | | Pricing model | Per committer, metered on test spans. Free up to 5 committers and 1M test spans per committer. Team price not published | Per committer, from $20 a month billed yearly ($29 on demand) | By test volume, from $99 a month for 3M tests. Included with BuildPulse Runners | Per active seat, $21 a month, with Merge Queue and CI Insights in the same plan. No test-volume metering. Free up to 5 contributors | ## Why teams pick Mergify And what you give up with each of the others. ### Choose Mergify Test Engine when - ✓ You want a flaky failure handled while the job is still running. Test Engine retries a known flaky test inside your test framework, so the build stays green without anyone clicking rerun. - ✓ You want new flaky tests stopped at the pull request. Prevention reruns the tests a pull request adds or changes and blocks the merge if one of them is flaky. - ✓ Flaky failures keep kicking pull requests out of your merge queue. Queue runs get the same retries and quarantine, and Mergify Merge Queue comes in the same plan. - ✓ You want quarantine to run itself: it waits for real evidence before quarantining a test and lets it out once it is healthy again. - ✓ You want one seat price for the merge queue, CI job health and test health, with no metering on test volume. [How Test Engine works](https://mergify.com/product/test-engine) ### Datadog Test Optimization: what you give up - Auto Test Retries reruns any failing test up to 5 times by default. A real bug that passes on one of those retries goes green. - A PR Gate cannot be made a required check on a GitHub repository that uses a merge queue, and there is no merge queue of its own. - Billing is per committer, on top of the rest of your Datadog contract. [Mergify vs Datadog Test Optimization](https://mergify.com/compare/datadog-test-optimization) ### Trunk Flaky Tests: what you give up - Trunk never retries a test itself. A flaky failure fails the job until the test is quarantined, so the first hits land on your developers and your merge queue. - It never reruns a new test either, so a new flaky test that passes once on its pull request merges unnoticed. - Quarantine works by having the Trunk CLI rewrite the exit code of every job that runs tests. - Usage is metered on test spans: the free plan stops at 5 committers and 1M spans per committer, and the paid price is not published. [Mergify vs Trunk](https://mergify.com/compare/trunk) ### BuildPulse: what you give up - Quarantine is a list served by an API: skipping those tests is wired into your framework by you. - No free plan, only a 14-day trial, and pricing climbs with test volume. No merge queue. [Mergify vs BuildPulse](https://mergify.com/compare/buildpulse) ## Where Mergify is different Test Engine runs inside your test framework, so it can act on a flaky failure while the job is still running. A test it already knows is flaky gets retried on the spot and the build stays green. Trunk and BuildPulse only see the results after the job has ended, so all they can do is quarantine the test once it has failed someone's build. Datadog retries inside the job too, but it retries every failing test, which also turns real bugs green when they pass on a second try. New flaky tests never get that far. Prevention reruns the tests a pull request adds or changes on the same commit and blocks the merge if one of them is flaky, while the author still has the context to fix it. For the rest, quarantine runs itself. Test Engine waits until a test has been unhealthy for a while, and ignores a run that turned many tests red at once, so an infrastructure hiccup does not quarantine half your suite. Once the test is healthy again it can leave quarantine on its own. All of it applies to merge queue runs, so with [Mergify Merge Queue](https://mergify.com/product/merge-queue) a flaky test never throws a pull request out of the queue. And it is one plan. Merge Queue, [CI Insights](https://mergify.com/product/ci-insights) and [Test Engine](https://mergify.com/product/test-engine) share a per-seat price with no metering on test volume. For the merge queue side of the market, see [how Mergify compares](https://mergify.com/compare-mergify). ## Frequently asked questions ### What is the best tool to detect flaky tests? Mergify Test Engine, because detection and remediation both happen inside your test framework. A known flaky test is retried in the same job, so the build stays green. A new flaky test is caught on its pull request and blocked before it merges. Whatever is left is quarantined automatically once the evidence holds and released when it is healthy again. It works on any CI, and Merge Queue and CI Insights come in the same seat price. Trunk and BuildPulse only work from the results you upload, and Datadog retries every failing test, real failures included. ### How do I quarantine flaky tests in CI? All four tools quarantine, and the difference is how much work it leaves to you. Mergify quarantines inside your test framework on any branch, decides on its own once the evidence holds, and releases the test when it is healthy again. Trunk's CLI rewrites the job's exit code when only quarantined tests failed. Datadog's libraries keep the test running but stop its failure from affecting CI status. BuildPulse serves a quarantine list through an API and your pipeline has to skip those tests itself. See [how to detect and quarantine flaky tests](https://mergify.com/guides/detect-and-quarantine-flaky-tests). ### Which tools catch a flaky test before it merges? Mergify and Datadog rerun new tests on purpose. Mergify's Prevention reruns the tests a pull request adds or changes, on the same commit, and blocks the merge if one is flaky, with or without a merge queue. Datadog's Early Flake Detection retries a test it has never seen up to 10 times, but its PR Gate cannot be required on a GitHub repository that uses a merge queue. Trunk and BuildPulse only read the results you upload, so they see flakiness when your CI happens to retry a test. ### Do flaky test tools work with JUnit XML? Yes. Mergify, Trunk and BuildPulse all take JUnit XML, so any framework that can write it is covered, and Datadog accepts it for test visibility. ### Do I need a merge queue to use a flaky test tool? No. All four work without one. Flaky tests just do the most damage in a queue, where one bad run can throw out a whole batch of pull requests. Mergify and Trunk both sell a merge queue next to their flaky test product. Datadog and BuildPulse do not. See [how the merge queues compare](https://mergify.com/compare-mergify). ### What's a good tool to manage my tests? If managing tests means knowing which ones you can trust, the four tools on this page are that category: they collect every test result from CI, separate flaky from broken, quarantine the noisy ones and track them until someone fixes them. Test case management tools for manual QA plans solve a different problem. ### How much do flaky test tools cost? Trunk bills per committer and meters test spans, free up to 5 committers and 1M spans per committer, with no published paid price. Datadog bills per committer, from $20 a month billed yearly. BuildPulse bills by test volume, from $99 a month for 3 million tests, and includes detection with its CI runners. Mergify bills $21 per active seat per month for Merge Queue, CI Insights and Test Engine together, with no metering on test volume, free for up to 5 contributors. [See Mergify pricing](https://mergify.com/pricing). ## Sources Every claim above comes from the vendor's own documentation or pricing page. - [Mergify: Test Engine](https://docs.mergify.com/test-engine) - [Mergify: Prevention](https://docs.mergify.com/test-engine/prevention) - [Mergify: Mitigation](https://docs.mergify.com/test-engine/mitigation) - [Mergify: Quarantine](https://docs.mergify.com/test-engine/quarantine) - [Mergify: Health status](https://docs.mergify.com/test-engine/health) - [Trunk: Flaky Tests](https://docs.trunk.io/flaky-tests) - [Trunk: Flake detection](https://docs.trunk.io/flaky-tests/detection) - [Trunk: Quarantining](https://docs.trunk.io/flaky-tests/quarantining) - [Trunk: New test monitor](https://docs.trunk.io/flaky-tests/detection/new-test-monitor) - [Trunk: Autofix flaky tests](https://docs.trunk.io/flaky-tests/agents/autofix-flaky-tests) - [Trunk: Pricing](https://trunk.io/pricing) - [Datadog: Flaky Test Management](https://docs.datadoghq.com/tests/flaky_management/) - [Datadog: Early Flake Detection](https://docs.datadoghq.com/tests/flaky_tests/early_flake_detection/) - [Datadog: Auto Test Retries](https://docs.datadoghq.com/tests/flaky_tests/auto_test_retries/) - [Datadog: PR Gates](https://docs.datadoghq.com/pr_gates/) - [Datadog: Pricing](https://www.datadoghq.com/pricing/?product=test-optimization) - [BuildPulse: Test quarantining](https://docs.buildpulse.io/flaky-tests/guides/Test%20Quarantining) - [BuildPulse: CI providers](https://docs.buildpulse.io/Guides/CI%20Providers/Intro) - [BuildPulse: Pricing](https://buildpulse.io/pricing) ## Stop flaky tests at the pull request. Test Engine, CI Insights and Merge Queue in one plan. Free for up to 5 contributors. [Get started](https://dashboard.mergify.com/)[See how Test Engine works](https://mergify.com/product/test-engine) --- Source: https://mergify.com/compare/github-merge-queue Mergify vs GitHub merge queue # You turned on GitHub's merge queue.Now one red run restarts the whole line. When a queued pull request fails, even on a flaky test, GitHub kicks it out and every run behind it starts over. Mergify tests pull requests in batches, finds the culprit when a batch fails, and keeps known flaky tests from ejecting anyone. [Start a free 14-day trial](https://dashboard.mergify.com/)[See the comparison](#comparison) Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama ## What GitHub does well, and where it stops | Feature | GitHub | Mergify | | --- | --- | --- | | Tests each PR against the latest base and the PRs ahead of it | ✓ | ✓ | | Several merge groups under test in parallel | ✓ | ✓ | | Merges several PRs at once | ✓ | ✓ | | Queue CI runs for 8 queued PRs | 8, one per PR | As few as 1, with batching | | When a run fails | Failing PR removed, rest rebuilt | Batch split to find the culprit | | Lighter checks on the PR, full suite in the queue | Same required checks for both | ✓ | | Urgent PRs | Jump to top, in-flight PRs rebuild | Priority levels by label or rule | | Monorepo scopes, parallel queues | – | ✓ | | Retry a job that failed for a transient reason | – | CI Insights | | Flaky tests quarantined so they stop ejecting PRs | – | Test Engine | | Queue pause and scheduled freezes | – | ✓ | | Queue analytics and event log | – | ✓ | | Slack and Datadog integrations | – | ✓ | | Price | Included with GitHub | Free for small teams, then per contributor | [Get started](https://dashboard.mergify.com/) Moved over from GitHub's merge queue - - - > "Mergify gives us the control and visibility we never had with GitHub's queue. It's become our CI command center." Ben Holtzman, Senior Design Enablement Engineer, [AheadComputing](https://mergify.com/customers/aheadcomputing) ## When GitHub's merge queue is enough Be honest with yourself. If these describe your team, the built-in option works fine: - ✓ Small team (fewer than 10 engineers) - ✓ Simple CI pipeline (single workflow, under 15 minutes) - ✓ Low PR volume (fewer than 20 per day) - ✓ No monorepo complexity - ✓ No need for CI batching to reduce costs ## When you need Mergify If any of these sound familiar, you've outgrown the built-in queue: - ✓ Team is growing (20+ engineers contributing daily) - ✓ CI is expensive and you need batching to reduce costs - ✓ Monorepo with multiple scopes and services - ✓ Hotfixes should automatically jump the queue based on labels or rules - ✓ Two-step CI: light checks on push, heavy checks in queue - ✓ Flaky tests are eating developer time - ✓ You need merge freezes and scheduling - ✓ CI runs exceed 30 minutes and reruns are costly > Early on, as the team, tests, and CI complexity grew, it was quite clear that we needed an automated tool to resolve bottlenecks. Our team loves the way Mergify solved our growing pains. Tomasz Biernacki Quality Assurance Engineer at Pitch ## What changes when you switch The table in pictures. Each panel is something you notice the first week, starting with what the queue sees. ### Live queue visibility What's running, what's waiting, and why, at a glance. #### GitHub Merge Queue Add metrics endpointqueued Fix billing edge casequeued Bump depsrunning Refactor schedulerqueued PR list, no context #### Mergify See the queue in real time ### Queue analytics Throughput, wait times, batch outcomes, CI failure rate, over time. #### GitHub Merge Queue Throughputn/a Avg. wait timen/a CI failure raten/a Batch outcomesn/a Queue healthn/a No queue analytics dashboard No historical view #### Mergify Full historical view ### Flaky test handling What happens when a flake takes down a green PR. #### GitHub Merge Queue PR enters queue CI runs Test flakes PR ejected from queue Author manually re-queues Flakes block your team #### Mergify PR enters queue CI runs Test flakes Auto-detect flake Quarantine + auto-retry Flakes don't block merges ### CI batching Test many PRs in one run, with bisect on failure. #### GitHub Merge Queue PR 1run #1: base + 1 PR 2run #2: base + 1 + 2 PR 3run #3: base + 1 + 2 + 3 3 CI runs One merge group run per PR #### Mergify PR 1 + PR 2 + PR 3 1 CI run Pass all 3 merge Fail bisect, retry rest 1 CI run One run for many PRs ### Priority handling Hotfixes should jump the line, predictably. #### GitHub Merge Queue Before \[A, B, C\] PR D jumps to the top After \[D, A, B, C\] In-flight A, B, C rebuild from scratch All-or-nothing #### Mergify Urgent PR High PRPR Default PRPRPR Low PR Routed by labels or rules Predictable order ### Monorepo / scope-aware queueing PRs in unrelated services shouldn't block each other. #### GitHub Merge Queue PRfrontend PRfrontend PRbilling PRbilling PRinfra PRinfra One linear queue Unrelated PRs block each other #### Mergify frontend PR PR billing PR PR infra PR PR Scope-aware, parallel queues Independent scopes run in parallel ### Monitoring and alerts Stream queue events to Slack, Datadog, and your own services. #### GitHub Merge Queue \# merge-queue No messages yet GitHub's queue does not post to Slack. It emits `merge_group` webhooks, and the rest is yours to build. Build the alerts yourself #### Mergify \# merge-queue MergifyAPP09:41 Merged 3 pull requests with 1 CI run #1421 reset password flow, #1422 reset endpoint, #1423 session expiry MergifyAPP10:12 Batch failed, culprit found #1430 left the queue (integration/checkout failed). #1428 and #1429 merged. MergifyAPP11:03 test\_invoice\_rounding quarantined Flaky on main. It still runs, it stops blocking merges. MergifyAPP17:00 main is frozen for the 2.4 release The queue keeps testing. Merges resume Monday at 09:00. Also streams queue metrics to Datadog and events to your own services Alerts that say what happened ### Merge integrity Your CI signs off on a SHA. The commit that lands on main should be that SHA. #### GitHub Merge Queue PR enters queue Stage merge commit SHA: a1b2c3d CI passes on a1b2c3d Server rebuilds merge SHA: 7d8e9f0 (new) Lands on main SHA: 7d8e9f0 Tested SHA ≠ Merged SHA #### Mergify PR enters queue Stage merge commit SHA: a1b2c3d CI passes on a1b2c3d same SHA Lands on main SHA: a1b2c3d Tested SHA = Merged SHA When the queue tests one SHA and lands a different one, branch protection still passes and the PR shows merged. You won't notice the drift until something breaks on main for reasons that don't match the diff. By then you're hunting through merge commits to figure out which ones were corrupted. This is not theoretical. [A widely-discussed Hacker News thread](https://news.ycombinator.com/item?id=47881672) walks through a real case where the native queue silently reverted merged code via this exact mechanism. > Mergify is a no-brainer. If you want to have your main branch always working then you have to introduce some merge queue functionality. Paco Sevilla DevOps Engineer at DeepDrive Switching ## Same checks, one file. The checks your branch protection requires today become the queue's merge conditions. Your workflows don't change. 1. Install the GitHub App on one repository. GitHub's queue keeps running the others. 2. Copy your required checks into `merge_conditions` in `.mergify.yml`. 3. Compare a week of CI runs. Queue analytics show runs per merged pull request. Switch the rest when the numbers convince you. ``` # .mergify.yml queue_rules: - name: default batch_size: 8 # one CI run can validate up to 8 PRs merge_conditions: - check-success = build - check-success = test scopes: source: files: api: include: - services/api/** web: include: - apps/web/** ``` ## Teams that merge more and break main less. [ ### 2x merge throughput with 3‑hour CI runs Read story → Cerebras 350 engineers · AI Infrastructure ](https://mergify.com/customers/cerebras) [ ### Cut CI time and kept main green Read story → Apex Fintech Solutions 350 engineers · Financial Technology ](https://mergify.com/customers/apex) [ ### 2x faster merges across a massive monorepo Read story → Back Market 300 engineers · E-commerce ](https://mergify.com/customers/back-market) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## FAQ ### Does Mergify replace GitHub's merge queue or work alongside it? It replaces it. Only one queue can own merging into a given branch, so pick Mergify or GitHub's native queue per repo. You can still run Mergify on some repos and GitHub's queue on others. ### Doesn't GitHub's merge queue already batch pull requests? Partly. GitHub can merge several pull requests at once (its merge limits go from 1 to 100), and each merge group includes the pull requests ahead of it. But it still dispatches one merge group, and so one CI run, per queued pull request. [GitHub's docs](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue) walk through it: pull request #2 gets its own branch containing the base, #1 and #2. Mergify can test a batch of pull requests in a single run, and splits the batch to find the culprit when it fails. ### Can I evaluate Mergify without disabling GitHub's merge queue? Yes. Install Mergify on a low-traffic repo first, leave GitHub's queue managing your busy repos, and compare for a couple of weeks. The [customer stories](https://mergify.com/customers) have a few examples of teams that did exactly that. ### Is the silently-reverted-code incident a real risk with GitHub's queue? Yes, and it stems from how the queue rebuilds merge commits server-side after CI passes. The merged SHA is not the SHA your CI signed off on, so a successful queue run can still land code that was not the code under test. [The original Hacker News thread documents the incident](https://news.ycombinator.com/item?id=47881672). Mergify does not rebuild the merge commit, so the tested SHA is the merged SHA. ### How long does it take to migrate from GitHub merge queue to Mergify? Most teams switch a single repo in under a day. Install the GitHub App, then commit a `.mergify.yml` with your queue rules. Larger setups with parallel queues take longer to design but are still day-one work, not week-one. ### Will my GitHub Actions workflows need to change? No. Mergify pushes each candidate merge to a temporary queue branch and opens a draft PR for it, so workflows you already trigger on push or pull\_request run there. If your existing PR CI works today, queue CI works without edits. The `merge_group` trigger GitHub's queue required can stay or go. ### How does pricing compare? GitHub merge queue is free, included with GitHub. Mergify is free for open-source projects and small teams; paid plans scale with active contributors. [See pricing](https://mergify.com/pricing) for current tiers. Last updated: October 14, 2026 ## Keep the queue. Lose the wait. All features included, up to 5 users. No credit card required. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/compare/graphite vs # Mergify vs Graphite Graphite and Mergify often show up in the same evaluation, even though they solve different problems. Here's an honest comparison. [Start a free 14-day trial](https://dashboard.mergify.com/)[See pricing](https://mergify.com/pricing) ## The short answer Graphite is a code review tool built around stacked diffs and AI review. Mergify is a CI platform with a merge queue, CI Insights, and Test Engine. They sit at different points in the same path, which is why plenty of teams run both: **Graphite for review, Mergify for everything between an approval and a green main.** ## Feature comparison | Feature | Graphite | Mergify | | --- | --- | --- | | Creating stacked PRs | Core product | – | | AI code review | Core product | – | | Merge queue | Team plan and up | Mature, since 2018 | | CI batching with bisect-on-failure | Private beta | ✓ | | Two-step CI | – | ✓ | | Monorepo / scope-aware queue | – | ✓ | | Flaky test detection + quarantine | – | Test Engine | | CI Insights (job health, auto-retry) | – | Full product | | Scheduled freezes and PR dependencies | – | Included | | Custom merge conditions (rule engine) | – | Full YAML rule engine | | Slack notifications (queue activity) | – | ✓ | | Datadog / monitoring integrations | – | ✓ | | Fast-forward merge | ✓ | ✓ | | Works with GitHub-native workflows | ✓ | ✓ | | Pricing model | Per user | Active contributors only | [Get started](https://dashboard.mergify.com/) [ Customer story 35% shorter feedback loop from PR to production. 60-engineer front-end team, single repo. PRs were waiting 4-5 hours to merge before they switched to Mergify. Read the full story → ](https://mergify.com/customers/productboard) Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama ## What each tool does best The two products lean in different directions. ### Graphite is great for - ✓ Stacked diffs as the daily review workflow - ✓ AI-assisted review on every PR - ✓ Fast review loops where each commit becomes its own focused review - ✓ Teams already running Graphite's CLI as part of their day-to-day ### Mergify is great for - ✓ Keeping main green at scale with a real merge queue - ✓ Monorepos with scope-aware parallel queues - ✓ Cutting CI cost via batching and two-step pipelines - ✓ Flaky test detection and quarantine without a separate product - ✓ Merge freezes and PR dependencies for governance - ✓ Landing GitHub stacked pull requests in order, each tested against real main ## Keep Graphite. Put a real queue under it. We are not going to tell you to rip out the tool your team reviews code in. Graphite is good at building and reviewing a stack, and since GitHub shipped native stacked pull requests the way you create one matters less every month. What does not get easier is the part after approval. Five approved pull requests still have to reach main in the right order, each tested against the state it will actually merge into, without one failure at the bottom quietly stranding everything above it. Worth being precise about how that works: Mergify's queue recognizes a stack from GitHub's own stacked pull requests, not from Graphite's branch chaining. Teams running both typically review in Graphite and stack on GitHub, which is what the queue then lands in order. So the two tools stack rather than compete: Graphite up to approval, Mergify's queue from approval to main, with the same batching, two-step CI and scope-aware lanes that every other pull request gets. ## Teams that merge more and break main less. [ ### 2x merge throughput with 3‑hour CI runs Read story → Cerebras 350 engineers · AI Infrastructure ](https://mergify.com/customers/cerebras) [ ### Cut CI time and kept main green Read story → Apex Fintech Solutions 350 engineers · Financial Technology ](https://mergify.com/customers/apex) [ ### 2x faster merges across a massive monorepo Read story → Back Market 300 engineers · E-commerce ](https://mergify.com/customers/back-market) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## FAQ ### Should I switch from Graphite to Mergify? For stacking and AI code review, no. Those are what Graphite is for, and Mergify does not replace either. The question worth asking is what happens after review: a stack of five approved pull requests still has to reach main in order without breaking it. That is a merge queue's job, and it is the layer most teams end up adding underneath Graphite rather than instead of it. ### Can I keep using Graphite and add Mergify? Yes, and that is the common setup. Keep reviewing in Graphite and put Mergify underneath for the merge: batching, two-step CI, scope-aware lanes and flaky-test handling all apply to your pull requests whatever created them. One thing to know if you want the queue to land a stack as an ordered unit: it reads stack membership from GitHub's own stacked pull requests, not from Graphite's branch chaining. Teams that want both usually review in Graphite and stack on GitHub. ### Does Graphite have a merge queue? Yes, on the Team plan and above. It runs CI on stacks in parallel and can fast-forward merge, and batching is in private beta. What it does not document is scope-aware lanes for monorepos, two-step CI, or flaky-test handling, which is usually why teams reviewing in Graphite put Mergify's queue underneath. ### How does pricing compare? Graphite charges per user: $20/user/month on Starter, $40 on Team, which is the first plan with the merge queue (both billed annually). Mergify charges per active contributor: anyone who opens a PR or pushes commits to one in a 30-day sliding window, prorated to the day. A 30-engineer team where 15 people merge in a given month: $600 on Graphite (or $1,200 with queue), $315 on Mergify Max. [See pricing](https://mergify.com/pricing). ### Will putting a Graphite stack through Mergify's queue break it? Nothing breaks: the pull requests queue and merge normally. They are simply not treated as one ordered unit, because Mergify reads stack membership from GitHub's own stacked pull requests rather than from Graphite's branch chaining. If you want the queue to land the chain in order, stack on GitHub and keep Graphite for review. ### Do Graphite users typically also adopt Mergify? Yes, especially as teams grow past simple branch-protection merging. Graphite handles the review side well, and once broken-main incidents start adding up, you want a queue underneath. Mergify is the typical answer for that layer. Last updated: September 29, 2026 ## Put a queue under the stacks you already build All features included, up to 5 users. No credit card required. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/compare/trunk # Mergify vs Trunk Both do merge queues. Trunk focuses on parallel testing and flaky test detection. Mergify gives you that plus CI observability, merge governance, a rule engine that handles your actual workflow, and monitoring integrations. [Start a free 14-day trial](https://dashboard.mergify.com/)[See pricing](https://mergify.com/pricing) ## Feature comparison | Feature | Trunk | Mergify | | --- | --- | --- | | Speculative checks | ✓ | ✓ | | CI batching with bisect-on-failure | ✓ | ✓ | | Bisection runs the tests that broke the batch first and stops early (Fail First) | – | ✓ | | Merge when a later batch proves an earlier failure was flaky | ✓ | ✓ | | Two-step CI | ✓ | ✓ | | Dynamic scopes from your build graph (Bazel, Nx, Turborepo, Pants) | ✓ | ✓ | | Direct merge when a PR's scopes are unaffected (no queue CI run) | ✓ | ✓ | | PR priorities | ✓ | Multiple levels | | Queue pause | ✓ | ✓ | | Queue freeze (scheduled) | – | ✓ | | Auto-retry on CI failure | Anti-flake only | Configurable rules | | Custom merge conditions (rule engine) | Status checks only | Full YAML rule engine | | Queue stats dashboard | ✓ | ✓ | | Slack notifications (CI + queue activity) | ✓ | ✓ | | Full event log | Per-PR only | Queue-wide | | Monitoring integrations (Datadog) | ✓ | ✓ | | CI Insights (job health, auto-retry) | – | Full product | | Flaky test detection + quarantine | Separate product | Full product (Test Engine) | | Scheduled freezes and PR dependencies | – | Included | | Billing by active contributors only | Per committer | Active contributors | | Fast-forward merge | – | ✓ | | On-premise deployment | Private preview (BYOC on AWS, Trunk-run control plane) | Mergify Enterprise Server (GHES or GitHub.com) | [Get started](https://dashboard.mergify.com/) ## What Trunk's comparison page gets wrong Trunk publishes its own Mergify comparison. Six of its claims do not match what Mergify does today, so here they are with the documentation. Checked 29 September 2026. Trunk says: “Flaky test protection: not in the queue.” Test Engine quarantines a flaky test so its failure no longer blocks the merge, on any branch, the queue included. The queue can also retry failed checks a set number of times before it gives up on a pull request. [Quarantine documentation](https://docs.mergify.com/test-engine/quarantine) Trunk says: “Optimistic merging: not offered.” With skip\_intermediate\_results, a pull request whose own batch failed still merges when a later batch containing its changes passes. A real bug would fail that larger batch too, so the earlier failure was noise. [Batches documentation](https://docs.mergify.com/merge-queue/batches) Trunk says: “Direct merge to main: not offered.” Mergify merges a queued pull request with no queue CI run when it is already up to date, or when main only moved in scopes it does not touch. Both cases are on by default. [Direct merge documentation](https://docs.mergify.com/merge-queue/direct-merge) Trunk says: “Dynamic parallel merge queues: manual, config required.” You switch on parallel mode once. From then on, batches that share a scope are chained, batches that share none are tested at the same time, and a failure in one lane does not hold up the others. Nobody draws the lanes by hand. [Queue modes documentation](https://docs.mergify.com/merge-queue/queue-modes) Trunk says: “Bazel integration: manual scopes only.” Your CI runs bazel-diff on every pull request and sends Mergify the impacted packages as scopes. Nx, Turborepo and Pants work the same way. Apex Fintech runs this on a Bazel monorepo with hundreds of engineers. [Bazel scopes documentation](https://docs.mergify.com/merge-queue/scopes/bazel) Trunk says: “GitHub outage resilience: webhook-driven, no recovery mechanism.” Webhooks are not the only input. Mergify re-evaluates every queued batch on a fixed schedule, so a batch whose GitHub events were lost recovers on its own, and checks\_timeout decides what happens when a CI provider goes quiet. [Queue lifecycle documentation](https://docs.mergify.com/merge-queue/lifecycle) You do not have to take our word for it. Base Power evaluated both of us while designing their own monorepo merge strategy, and wrote up what they found:[“we evaluated trunk.io and Mergify, which both offer parallel merge queues with dynamically configured lanes”](https://inside.basepowercompany.com/p/merge-dont-queue). Used by the platform teams that ship fast and keep CI green "Two-step CI completely transformed how we handle long hardware tests. Instead of burning hours on repeated runs, we validate once and merge with confidence." Rushin Shah, Sr. Manager DevInfra · Cerebras "The merge queue is a huge value from Mergify: it keeps our main stable and allows us to merge quickly with confidence." Minh Tran, Platform Engineer · Apex Fintech Solutions "You get the benefits of Mergify immediately. There is no competitor doing stuff like this." Florian Berchtold, Software Engineer · Luminar "The merge queue is a time-saver! Contributors no longer have to carry the mental load of monitoring their PRs!" Johan Lorenzo, Staff Release Engineer · Mozilla "We transitioned from manual chaos to over 20 safe deploys per day. Mergify is becoming a critical part of our delivery function." Chris Irving, Engineering Manager · Jane "Our merge queue is like a safety net. Every PR that lands has been tested, approved, and verified under the exact same conditions." Fabien Marty, Senior Staff Software Engineer · Botify "With dozens of engineers and long-running CI jobs, Mergify turned what used to be a daily bottleneck into a background process. Our teams focus on hardware, not merge logistics." Ben Holtzman, Senior Design Enablement Engineer · AheadComputing "Before Mergify, we spent too much time manually coordinating merges. Now everything flows automatically." Nicolas Beaussart-Hatchuel, Senior Staff Software Engineer · PayFit "Mergify was the best fit. It saved a couple of hours per engineer per month and resolved our bottleneck with the merge queue." Tomas Nekolny, Engineering Manager · Productboard "The support on Slack is really good, reactive and transparent. We have other partners where problems feel less transparent. With Mergify, it's fluid." Clément Danjou, VP Blockchain · Zama ## More than a merge queue Trunk sells a merge queue and a flaky test tool as separate products. Mergify includes merge queue, CI observability, test intelligence, and merge governance in every plan. ✓ Three products in one plan: Merge Queue, CI Insights, and Test Engine. Trunk sells merge queue and flaky tests separately. ✓ Full YAML rule engine for merge conditions based on labels, files, authors, time windows, and custom logic ✓ Scheduled freeze windows for releases, holidays, deploy cycles, and maintenance periods ✓ CI Insights gives you job health, failure rates, auto-retry with custom rules, and Slack alerts. Test Engine adds flaky detection, quarantine, and test-level metrics. ✓ Full queue-wide event log for auditing and debugging what happened and when ✓ Running since 2018 with 8 years of production hardening across thousands of repos ## Evaluating merge queue tools? We'll walk you through how Mergify compares for your specific setup. No pitch, just answers. [Book a discovery call](https://mergify.com/discovery) > Early on, as the team, tests, and CI complexity grew, it was quite clear that we needed an automated tool to resolve bottlenecks. Our team loves the way Mergify solved our growing pains. Tomasz Biernacki Quality Assurance Engineer at Pitch ## What Trunk doesn't cover Trunk is strong on merge queues and flaky tests. These are the gaps teams notice when they need more from their CI tooling. ### No scheduled freeze windows Trunk can pause the queue, but has no scheduled freeze windows for releases, holidays, deploy cycles, or recurring maintenance. ### No scheduled freezes or PR dependencies No equivalent to Mergify's scheduled freezes, time-based merge windows, or Depends-On for coordinating pull requests across repositories. ### No CI Insights Trunk has a flaky test product, but no CI-wide observability. No job health scores, no failure rate trends, no auto-retry with custom rules. Mergify covers both with CI Insights and Test Engine. ### No configurable retry rules Trunk retries implicitly through anti-flake detection. You can't set custom rules like 'retry this job up to 3 times if it matches this pattern.' ### No event log or audit trail No full event log for queue activity. When something goes wrong, there's no trace to investigate. ### No fast-forward merge Trunk doesn't support fast-forward merge. If you need to preserve commit SHAs and linear history without recreating commits, Mergify can do this. > Mergify is a no-brainer. If you want to have your main branch always working then you have to introduce some merge queue functionality. Paco Sevilla DevOps Engineer at DeepDrive ## The pricing question Trunk charges per committer for their merge queue. They sell flaky test detection as a separate product with its own pricing. Mergify charges $21/user/month and includes merge queue, CI Insights (job health, auto-retry), and Test Engine (flaky detection, quarantine) in a single plan. Both offer free tiers. Mergify is free for up to 5 users on private repos. Trunk is free for up to 5 committers. [See Mergify pricing](https://mergify.com/pricing)[Book a discovery call](https://mergify.com/discovery) ## Teams that merge more and break main less. [ ### 2x merge throughput with 3‑hour CI runs Read story → Cerebras 350 engineers · AI Infrastructure ](https://mergify.com/customers/cerebras) [ ### Cut CI time and kept main green Read story → Apex Fintech Solutions 350 engineers · Financial Technology ](https://mergify.com/customers/apex) [ ### 2x faster merges across a massive monorepo Read story → Back Market 300 engineers · E-commerce ](https://mergify.com/customers/back-market) [ ### 100% green main, faster shipping Read story → PayFit 150 engineers · HR Tech ](https://mergify.com/customers/payfit) ## FAQ ### Does Mergify replace Trunk or work alongside it? It replaces Trunk's merge queue. Only one queue can own merging into a given branch, so you pick Mergify or Trunk per repo. You can run Mergify on some repos while you evaluate. ### What does Mergify include that Trunk sells as a separate product? Trunk sells its merge queue and its flaky test tool as two products with separate pricing. Mergify includes the merge queue, CI Insights for job health and auto-retry, and Test Engine for flaky detection and quarantine in every plan. ### Can Mergify schedule merge freezes for releases or holidays? Yes. Trunk can pause the queue on demand, but it has no scheduled freeze windows for release cycles, holidays, or recurring maintenance. Mergify schedules freezes ahead of time and keeps an exception path for hotfixes. See [how to run a merge queue during a code freeze](https://mergify.com/guides/merge-queue-code-freeze). ### Does Mergify support fast-forward merge? Yes. If you need to preserve commit SHAs and keep a linear history without recreating commits, Mergify can fast-forward merge. Trunk does not offer this. ### How long does it take to migrate from Trunk to Mergify? Most teams switch a single repo in under a day. Install the GitHub App, then commit a `.mergify.yml` with your queue rules. Larger setups with parallel queues take longer to design but stay day-one work. ### How does pricing compare? Trunk charges per committer for the merge queue and prices flaky test detection separately. Mergify is $21 per user per month with the merge queue, CI Insights, and Test Engine in one plan. Both are free for up to 5 users. [See pricing](https://mergify.com/pricing). Last updated: September 29, 2026 ## Try the complete CI platform Merge queue, CI Insights, and Test Engine. All included, up to 5 users free. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/guides Guides # How to actually do it. Short, task-focused how-tos for the day-to-day work of running a merge queue and keeping CI fast. Each one ships with a config you can copy and adapt. ## All guides [Guide ### How to set up a merge queue on GitHub Install the Mergify app, commit a small .mergify.yml, and queue your first pull request. With a validated config you can copy. Read the guide →](https://mergify.com/guides/setup-github-merge-queue)[Guide ### How to find your slowest CI jobs with CI Insights Rank your jobs by the time they actually cost, and tell a slow job apart from a slow runner. Read the guide →](https://mergify.com/guides/find-slowest-ci-jobs)[Guide ### How to cut CI costs with batching and bisection Batch several pull requests into one CI run and bisect only when it fails, with the math for picking a batch size. Read the guide →](https://mergify.com/guides/cut-ci-costs-batching-bisection)[Guide ### How to detect and quarantine flaky tests automatically Send your test results to Mergify, let it classify which tests are flaky, and turn on auto-quarantine so their failures stop blocking merges. Read the guide →](https://mergify.com/guides/detect-and-quarantine-flaky-tests)[Guide ### How to manage dependent PRs with Depends-On Declare pull request dependencies with a Depends-On header, and Mergify holds the merge until the ones it names have landed. Read the guide →](https://mergify.com/guides/manage-dependent-prs-depends-on)[Guide ### How to run a merge queue during a code freeze Schedule a freeze and carve out the hotfix path, while the queue keeps testing pull requests until you lift it. Read the guide →](https://mergify.com/guides/merge-queue-code-freeze)[Guide ### How to stop flaky tests from blocking your merge queue Anti-flake queue settings, automatic job retries, and quarantine, so one unreliable test stops costing you a queue full of merges. Read the guide →](https://mergify.com/guides/stop-flaky-tests-blocking-merge-queue)[Guide ### How do I test AI-generated pull requests before they merge? Keep your required checks, then test each pull request against the main it will land on. Batching for volume, Prevention for new flaky tests. Read the guide →](https://mergify.com/guides/test-ai-generated-pull-requests)[Guide ### Do I need a merge queue if my CI is fast? Fast CI shortens each run, and two green pull requests can still break main. Merge skew explained, and how to tell whether you need a queue. Read the guide →](https://mergify.com/guides/merge-queue-if-ci-is-fast)[Guide ### How do I merge pull requests from Claude Code and other coding agents safely? What happens after the agent opens the pull request: CI that actually runs, a human approval, the queue, and what the agent does when a check fails. Read the guide →](https://mergify.com/guides/merge-pull-requests-from-coding-agents)[Guide ### How do I debug flaky tests and intermittent CI failures? Rerun on the same commit, compare a failing run with a passing one, and match the difference to a known cause. Repeat commands and a symptom table. Read the guide →](https://mergify.com/guides/debug-flaky-tests-intermittent-ci-failures) More guides on the way. For the concepts behind them, see [Learn](https://mergify.com/learn). --- Source: https://mergify.com/guides/cut-ci-costs-batching-bisection Guide # How to cut CI costs with batching and bisection You cut CI costs by testing several pull requests in one CI run instead of one run each, and bisecting only on the days a run fails. Set `batch_size` on your queue rule and twenty merges a day stop costing twenty full pipelines. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated July 29, 2026 ## In one paragraph Batching combines the next few queued pull requests into one test branch and runs your pipeline once against the combined change. If it passes, they all merge. If it fails, Mergify splits the batch and retests the parts until it finds the pull request that broke it, drops that one, and merges the rest. The saving is large, and bisection costs a handful of extra runs on the days something breaks. The table below prices that in. ## Where does the queue CI budget actually go? A [merge queue](https://mergify.com/learn/merge-queue) that validates one pull request at a time runs your full pipeline once per merge, on top of the run that pull request already triggered when it was pushed. Twenty merges a day against a 30-minute suite is ten hours of runner time, every day, spent re-testing code that has already been tested once. Most advice about CI costs works on the inside of a run: cache dependencies, move to a cheaper runner class. Those help, and they cap out at a percentage of each run. Batching deletes runs instead of shrinking them, which is a different order of magnitude, and it is available to any team already running a queue. ## How much does batching actually save? Take a team merging 20 pull requests a day, a 30-minute queue pipeline, and one pull request in twenty that fails in the queue. That last number is the one everybody forgets, so the bisection runs are counted here as a cost, not waved away. | batch\_size | Batches | Bisection runs | Total runs | CI hours/day | Saved | | --- | --- | --- | --- | --- | --- | | 1 (no batching) | 20 | 0 | 20 | 10.0 | baseline | | 3 | 7 | 2 | 9 | 4.5 | 55% | | 5 | 4 | 3 | 7 | 3.5 | 65% | | 10 | 2 | 4 | 6 | 3.0 | 70% | The first step is the one that pays. Going from no batching to batches of three removes more than half the queue's CI time. Doubling from five to ten buys another five percentage points while making every failure slower to resolve and every merge wait longer for a batch to fill. Most teams land between three and five and stop there. The bisection column assumes one failing batch a day and a binary split, which is what Mergify does by default. If your queue failure rate is closer to one in five than one in twenty, redo the arithmetic with more failing batches before you pick a size. ## What happens when a batch fails? A failing batch does not send five people back to their branches. Mergify splits it and retests cumulative parts of it, using the failure it already knows about to skip combinations it has ruled out: ``` Batch [#1, #2, #3, #4, #5] fails → test [#1, #2] and [#1, #2, #3, #4] in parallel → [#1, #2] passes, [#1, #2, #3, #4] fails → the culprit is #3 or #4 → test [#1, #2, #3] → fails → #3 is the culprit and leaves the queue → #1 and #2 merge, #4 and #5 go back in the queue ``` The split runs in parallel up to `max_parallel_checks`, so raising that value trades CI concurrency for a faster recovery. If a part contains a single pull request and still fails, that pull request is the culprit and it is removed with a comment explaining why. Everything else keeps its place in the queue. Two knobs matter when bisection gets expensive. `batch_max_failure_resolution_attempts` caps how many times a batch may be split before the whole thing is dropped from the queue. `skip_intermediate_results` stops a flaky intermediate failure from blocking a pull request whose changes later pass inside a bigger batch, which is worth turning on if your suite is not perfectly stable. ## How do I turn it on? Batching is one key on a queue rule. This config adds it to a queue that also splits validation into a cheap entry check and an expensive pre-merge suite, which is where the two savings compound: ``` merge_queue: # How many batches (or batch splits) run at the same time. max_parallel_checks: 2 queue_rules: - name: default # Cheap checks decide who gets into the queue. queue_conditions: - check-success=lint-and-unit # The expensive suite runs once per batch, not once per pull request. merge_conditions: - check-success=full-suite batch_size: 5 batch_max_wait_time: 5 min merge_method: squash ``` - `batch_size` is how many queued pull requests are tested in one run. This is the setting that removes runs from your bill. - `batch_max_wait_time` is how long Mergify waits for a batch to fill before testing a smaller one. Without it, a quiet afternoon produces batches of one and the saving disappears. - `max_parallel_checks` is how many batches or splits run at once. It buys throughput and recovery speed, and it costs concurrency. - `queue_conditions` and `merge_conditions` are the two-step split: lint and unit tests gate entry, and the full suite runs only against the batch that is about to land. If you do not have a queue yet, start with [setting up a merge queue on GitHub](https://mergify.com/guides/setup-github-merge-queue) and add batching once pull requests are flowing through it. ## What batch size should I pick? Pick from your queue failure rate: - **Under 2% of queued pull requests fail:** batches of 5 to 10. Bisection is rare enough that its cost rounds to nothing. - **2% to 5%:** batches of 3 to 5. This is where most teams sit. - **Over 5%:** batches of 2 to 3, and treat the failure rate as the real problem. At a high enough rate the bisection runs cancel the saving. The practical signal is easier than the percentage: if bisection fires more than once a day, your batch is too big or your tests are too flaky. Start at 3, watch a week of queue activity, and move up. One thing that is not a reason to keep batches small is fear of unrelated changes colliding. Mergify groups pull requests that touch the same scopes and directories, so a batch tends to be a set of related changes rather than a random slice of the queue. The [batching explainer](https://mergify.com/learn/merge-queue/batching) covers the trade-offs in more depth. ## What else takes runs off the bill? Batching is the biggest single lever, and it stacks with a few others. - [Two-step CI](https://mergify.com/learn/merge-queue/two-step-ci) keeps the expensive suite off pull requests that are still being reviewed. Combined with batching, a 30-minute suite runs once for five pull requests instead of five times per pull request per push. - Scopes tag each pull request with the areas of the repository it touches, so [monorepo CI](https://mergify.com/learn/monorepo/ci) can run only the jobs those areas need. A documentation change stops paying for the integration suite, and the queue reuses the same scopes to build better batches. - Flaky tests are a spending problem before they are a trust problem. Every retry is a full pipeline you paid for twice. [CI Insights](https://mergify.com/product/ci-insights) shows which jobs and tests are burning runs, which is usually a shorter list than people expect. None of this requires rewriting your pipeline. Batching, two-step CI, and scopes are all queue configuration, and they work with whatever CI you already run. For the broader picture of what a queue does to your pipeline, see [continuous integration](https://mergify.com/learn/continuous-integration). ## FAQ ### Does batching mean a broken pull request can reach main? No. A batch is tested as a whole before anything merges, so a change that breaks the build fails the batch and never lands. What batching does hide is a failure that only shows up in one pull request in isolation and is masked by another change in the same batch. That is rare, and it is the trade you accept for running one pipeline instead of five. ### How does Mergify decide which pull requests go in a batch together? It groups pull requests that touch similar parts of the repository, ranked by shared scopes first, then by the directories they change, then by queue time. Priority and stacks come first: a higher-priority pull request is never held back for similarity, and a stack always stays in one batch. Grouping related changes makes a failed batch cheaper to resolve. ### What happens to the pull requests behind a failing batch? They are cancelled and put back in the queue in the same order. Once the culprit is isolated and removed, they are retested against the new state of the branch and merge normally. Nobody has to re-queue anything by hand. ### Our CI is flaky. Should we still batch? Batch smaller, and turn on skip\_intermediate\_results. With that option, a pull request whose changes are contained in a later passing batch is not blocked by an earlier failing one, so a transient failure stops costing you a full bisection. Fix the flaky tests too, because at a high failure rate the bisection runs eat the saving. ### Do we need bigger CI runners for batches? Sometimes. A batch of five pull requests is one branch with five changes on it, so the test suite runs once over a slightly larger diff. Test time usually barely moves, but if your suite is sharded by changed files or your runners are already at their memory ceiling, check that before you set batch\_size to 10. ### Can we cap how much bisection is allowed to cost? Yes. batch\_max\_failure\_resolution\_attempts limits how many times Mergify splits a failing batch. When the limit is hit, the whole batch leaves the queue instead of being narrowed down further. Use it if you would rather pay a human to look at a bad day than pay for the runs. Full reference: [batches](https://docs.mergify.com/merge-queue/batches) and [performance tuning](https://docs.mergify.com/merge-queue/performance). Last updated: July 29, 2026 ## Stop paying for one pipeline per pull request. Add a batch size to your queue and let Mergify bisect the failures. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/guides/debug-flaky-tests-intermittent-ci-failures Guide # How do I debug flaky tests and intermittent CI failures? Rerun the failing test on the same commit. If the result changes, the code did not cause it, so compare a failing run with a passing one, match the difference to a known cause, and start with the test that costs your team the most. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 4, 2026 ## In one paragraph A rerun tells you a test is flaky and nothing about why, which is how a suite ends up with a test everybody reruns and nobody fixes. Debugging starts once you hold a red run and a green run of the same code, because the difference between them (order, workers, timing, runner) usually names the cause. ## Is it flaky, or did my change break it? Run it again on the same commit. Two results on one commit mean the outcome does not follow from the code. If it fails twice, run it on the parent commit too: a failure there means your change is not the cause, and a pass means it probably is. One rerun is a weak test of a failure that shows up one run in twenty, so repeat it until you have a failure to look at: ``` # Go: 100 runs in shuffled order, with the race detector go test -run '^TestCheckout$' -count=100 -shuffle=on -race ./... # pytest, with the pytest-repeat plugin pytest tests/test_checkout.py --count=50 # Playwright npx playwright test tests/checkout.spec.ts --repeat-each=50 ``` ## What does the failure pattern point to? | What you see | Usual cause | What to do next | | --- | --- | --- | | Passes alone, fails in the full suite | Test order, or state left behind by an earlier test | Shuffle the order, then cut the tests that run before it in half until the failure goes away. | | Fails only with parallel workers | A shared database, port or temp directory | Run with one worker to confirm, then give each worker its own copy. | | Fails at certain hours or dates | The clock, a time zone or a month boundary | Freeze the clock in the test and run with the runner's time zone. | | Times out on CI, passes on a laptop | A fixed sleep or an unawaited call on a slower machine | Wait for a condition instead of a duration, and limit CPU locally to reproduce it. | | The job dies and no test failed | The runner ran out of memory, or the network or a service dropped | Read the job log around the exit. There is no test to fix here. | Each framework has its own favorite version of these. The [flaky tests guide](https://mergify.com/learn/flaky-tests) has breakdowns for [pytest](https://mergify.com/learn/flaky-tests/pytest), [Jest](https://mergify.com/learn/flaky-tests/jest), [Playwright](https://mergify.com/learn/flaky-tests/playwright), [Go](https://mergify.com/learn/flaky-tests/go) and more. ## What should I compare between a failing and a passing run? Open both logs side by side. Start with the random seed if your runner prints one, since replaying it can turn an intermittent failure into a repeatable one. Then check: - the test order, and which worker ran the test - the runner type and image, which can change under a floating tag - dependency versions, if anything installs without a lockfile - stderr and timing around the failure, including the test that ran just before it ## Is it a flaky test or a flaky job? A job can fail with every test green, and then the test logs will not help you. [Test Engine](https://mergify.com/product/test-engine) tracks health per test, and `mergify tests show ''` prints a test's health, ratios and last failure from your default branch. In a script, read `.tests[].health_status` from `--json`, since the command exits 0 on a flaky test too. [CI Insights](https://mergify.com/product/ci-insights) gives [each job](https://docs.mergify.com/ci-insights/jobs) a health status from its recent runs, which is where a runner or setup problem shows up. ## Which flaky test do I debug first? The one that fails most often in a required check, because that is the one stopping people from merging. A test that fails weekly in a nightly job can wait. [Detecting and quarantining flaky tests](https://mergify.com/guides/detect-and-quarantine-flaky-tests) shows how to get the full list without collecting it by hand. Quarantine the test while you work on it. It keeps running and its failures [stop blocking merges](https://mergify.com/guides/stop-flaky-tests-blocking-merge-queue), so nobody has to rerun around it during the days the fix takes. Rerunning instead [gets the build green and fixes nothing](https://mergify.com/blog/re-run-failed-jobs-is-not-flaky-test-management). ## How do I know the fix worked? Rerun the repeat command, with more runs than feels necessary. A test that fails one run in twenty still passes 20 runs in a row about a third of the time. At 100 runs, the odds drop below 1%. Then watch it on your default branch. Test Engine [calls a flaky test healthy again](https://docs.mergify.com/test-engine/health) only after a streak of clean passes over several days, because a flaky test can pass for a while and fail again. Turn on [`unquarantine_when_healthy`](https://docs.mergify.com/test-engine/quarantine) and the test leaves quarantine once it gets there. ## FAQ ### What is the difference between a flaky test and a broken test? A flaky test's result does not follow from the code: it passed and failed on the same commit, for example. A broken test fails and nothing points to flakiness, so the code or the test is simply wrong. They need different work, which is why it pays to tell them apart before debugging. ### Should I just retry failed tests? A retry gets one pull request moving and leaves the test as flaky as it was. It is a reasonable way to keep working while someone fixes the test, and a bad place to stop, because the next person hits the same failure. ### Why does a test only fail in CI? CI machines usually have less CPU, a different number of workers, a clean filesystem and a different time zone from your laptop. Any of those can expose a race or a hidden assumption, so run the repeat command inside a CI job when it will not reproduce locally. ### Should I delete a flaky test? If nobody can say what it protects, yes. A test that fails at random and covers something already tested elsewhere costs more than it catches. Otherwise quarantine it while you work on the fix, so the coverage stays and merges keep moving. Full reference: [test health](https://docs.mergify.com/test-engine/health), [detection](https://docs.mergify.com/test-engine/detection) and [quarantine](https://docs.mergify.com/test-engine/quarantine). Last updated: November 4, 2026 ## See which tests are flaky before you start debugging. Test Engine classifies every test as healthy, flaky or broken from its history, and quarantines the ones blocking merges. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how Test Engine works](https://mergify.com/product/test-engine) --- Source: https://mergify.com/guides/detect-and-quarantine-flaky-tests Guide # How to detect and quarantine flaky tests automatically Upload your test results to Mergify, let it classify each test from its own history, then turn on auto-quarantine so flaky tests stop failing your pipeline while they keep running and reporting. No test code changes, no manual list to maintain. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated July 29, 2026 ## In one paragraph Add one upload step to your CI job so every JUnit report reaches Mergify, including the runs that fail. After a few days, Detection lists the tests that pass and fail on the same commit and ranks them by how much CI time they waste. Enable auto-quarantine on the repository and Mergify takes the flaky ones out of the way as it finds them: they still run, their results still land in the dashboard, their failures just stop deciding whether your pipeline is red. ## What you need before you start - A test suite that can write a JUnit XML report. Every major runner does this, usually behind one flag. - A Mergify account with the GitHub App installed, and a `MERGIFY_TOKEN` stored as a CI secret. You do not need to touch a single test. Detection reads the reports your CI already produces, and quarantine is a decision applied on Mergify's side, not an annotation in your code. If you want the background on why quarantine beats skipping or deleting, the [test quarantine guide](https://mergify.com/learn/test-quarantine) covers the tradeoffs. ## Step 1: Send your test results to Mergify Detection needs the failures, so the upload has to happen whether the suite passed or not. In GitHub Actions that means two changes to the job that runs your tests: ``` - name: Run tests id: tests continue-on-error: true run: pytest --junit-xml=report.xml - name: Mergify CI Upload if: success() || failure() uses: Mergifyio/gha-mergify-ci@v25 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: report.xml test_step_outcome: ${{ steps.tests.outcome }} ``` `continue-on-error: true` on the test step is what makes quarantine possible later. It hands the pass or fail decision to the Mergify step, which is the only step that knows which failures came from quarantined tests. Passing `test_step_outcome` covers the nastier case where the runner crashes and leaves a report that looks clean. Pytest users can skip the report plumbing entirely: the `pytest-mergify` plugin uploads results on its own once `MERGIFY_TOKEN` is in the environment. Buildkite and the language-specific setups are in the [Test Engine docs](https://docs.mergify.com/test-engine). ## Step 2: Let detection classify your tests Once reports are flowing, Mergify classifies each test from its results across runs, weighting recent ones more heavily. Only the tests behaving badly show up: | Status | What it means | What to do | | --- | --- | --- | | Flaky | Passes and fails on the same commit, with no code change between runs. | Auto-quarantine handles these by default. Fix them in impact order. | | Broken | Fails consistently, with recent runs weighted more heavily than old ones. A test that started failing this week is broken, not flaky. | Quarantine is opt-in here, because a broken test is usually telling you something true. Turn it on when broken tests keep the pipeline red. | | Healthy | Nothing unusual in the recent history. | Not listed in Detection at all. You only ever look at the tests that misbehave. | Two numbers sit next to each test. Confidence says how much data Mergify has: low confidence means the verdict could still move, so give a fresh setup a few days before you act on it. Impact counts the failed executions a test caused, which is the honest measure of what it costs you. Sorting by impact usually shows the same thing in every repository, that a handful of tests are responsible for most of the red builds. If you would rather fix than quarantine, the framework-by-framework breakdowns in the [flaky tests guide](https://mergify.com/learn/flaky-tests) name the usual culprits: shared state across parallel workers, frozen clocks, mocks that outlive their test, ordering assumptions that only hold on your laptop. [Debugging flaky tests and intermittent CI failures](https://mergify.com/guides/debug-flaky-tests-intermittent-ci-failures) walks through finding which one you have. ## Step 3: Turn on auto-quarantine Auto-quarantine is a per-repository switch on the Mitigation page of the [dashboard](https://dashboard.mergify.com/). Enable it and Mergify quarantines flaky tests as detection finds them, without anyone filing a ticket first. Quarantining known-broken tests is a second option next to it, deliberately separate: a consistently failing test is usually reporting something real, and you should decide to ignore it rather than drift into it. A quarantined test still runs and still shows its full failure history in the dashboard. What changes is the conclusion your CI reports. Quarantine applies on every branch, not only the default one, so a pull request author gets the benefit immediately. When you want to handle one test by hand, the CLI does it from the terminal or from a script: ``` # Quarantine a test by its fully qualified name. mergify tests quarantines add \ --reason "flaky under xdist, tracked in MRGFY-1234" \ "tests/auth/test_login.py::test_login_timeout" # See everything currently quarantined, and why. mergify tests quarantines list # Take it back out once it is fixed. mergify tests quarantines remove \ "tests/auth/test_login.py::test_login_timeout" ``` The same commands are worth wiring into an AI coding agent. `mergify tests show` returns a test's health as an exit code, so an agent that hits a red test can ask whether it is flaky before it spends an hour rewriting code that was never broken. ## What this changes in your merge queue Nothing in your queue config, which is the point. The queue keeps gating on the same CI check it always did: ``` queue_rules: - name: default merge_conditions: - check-success=ci pull_request_rules: - name: Queue approved pull requests conditions: - "#approved-reviews-by>=1" actions: queue: name: default ``` That check is now reported by the Mergify upload step, so it goes green when the only failures came from quarantined tests. A flaky test that used to knock a pull request out of the queue, send it to the back, and burn a full CI run on the retry stops doing any of that. If you have not set the queue up yet, the [merge queue guide](https://mergify.com/learn/merge-queue) walks through how it works and how to configure one. Quarantine also pairs with [auto-retry](https://mergify.com/learn/auto-retry): retry buys you a second opinion on a suspicious failure, quarantine is what you reach for once the test has proven it cannot give one. ## Step 4: Keep the quarantine list honest The failure mode of any quarantine system is that it becomes a graveyard. Tests go in, nobody looks again, and a year later a quarter of the suite is decorative. Two habits prevent it. Put a real reason on every manual quarantine, with a ticket reference. Six months later "flaky" tells nobody anything, while "flaky under xdist, tracked in MRGFY-1234" tells them where to look. Then review the list on a fixed cadence, monthly is enough for most teams. Mergify records whether each quarantined test has recovered, which turns the review into a short list of tests you can take back out rather than an audit of everything. Watch the trend too. A quarantined test that goes from failing 5% of the time to 40% is telling you something changed, even though it is no longer blocking anyone. That is the part [CI Insights](https://mergify.com/product/ci-insights) is for: quarantine removes the interruption, the data stays. ## FAQ ### How does Mergify know a test is flaky? From its history, not from a single run. Mergify compares results for the same commit across CI runs: a test that passes on one run and fails on another without a code change is classified as flaky. A test that fails consistently, with recent runs weighted more heavily, is classified as broken instead. ### Does a quarantined test stop running? No. It runs exactly as before and its results keep flowing into Test Engine. The only thing that changes is that its failures no longer mark the pipeline as failed, so they no longer block a merge. ### How long before detection has enough data? It depends on how often your suite runs. Each test carries a confidence level that reflects how many runs Mergify has collected for it. Treat low-confidence results as preliminary and give a new setup a few days of normal traffic before acting on them. ### Can I quarantine a test from a script or an AI coding agent? Yes. The Mergify CLI has mergify tests quarantines add, remove, and list, and mergify tests show reports a test's health through its exit code (0 healthy, 1 flaky, 6 broken). An agent that hits a failing test can check the health first and quarantine it instead of chasing a phantom regression. ### Will auto-quarantine hide a real regression? It can, which is why only flaky tests are quarantined automatically by default and quarantining broken tests is a separate option you turn on deliberately. Every quarantine keeps its reason and its history, and recovered tests are flagged, so the list stays reviewable rather than becoming a place tests go to die. ### Does this work outside GitHub Actions? Yes. Buildkite has its own integration, and the Mergify CLI uploads a JUnit report from any CI that can run a command. The detection and quarantine behaviour is the same whichever way the results arrive. Last updated: July 29, 2026 ## Stop chasing flaky tests Upload your test results, see which tests are actually flaky, and let Mergify quarantine them. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how CI Insights works](https://mergify.com/product/ci-insights) --- Source: https://mergify.com/guides/find-slowest-ci-jobs Guide # How to find your slowest CI jobs with CI Insights Enable CI Insights on the repository, open the Jobs page, and sort by duration over a 30-day window. Then multiply duration by how often each job runs. The job at the top of that second ranking is your real bottleneck, and it is usually not the one people complain about. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated July 29, 2026 ## In one paragraph CI Insights reads job data straight from GitHub Actions, Jenkins, or Buildkite, so the Jobs page fills in without you touching a workflow file. Sort by p95 duration to find the runs that hurt, then rank by duration times executions to find where the hours actually go. Check the Runners page to see how much of that time was spent waiting for a machine rather than running your code. Then cut the jobs that never needed to run in the first place. ## Why does the pipeline feel slow when every job looks fine? A pipeline finishes when its longest chain finishes, so the average job duration tells you nothing useful. Nine jobs at four minutes and one at forty gives you a forty-minute pipeline and a very reasonable-looking average of about eight. Ask most teams which job is the slow one and you will get an answer, confidently, and it will be the job somebody watched fail at 6pm last Thursday. There are two other numbers folded into the one your CI shows you. The first is wait: the minutes a job spent queued for a runner before a single line of your code executed. The second is frequency. A nine-minute job triggered on every pull request costs your organization more hours per week than a nightly that takes an hour, and unlike the nightly it sits directly between an engineer and their merge. Ranking jobs correctly takes about ten minutes once the data is there. For the concepts underneath, the [continuous integration guide](https://mergify.com/learn/continuous-integration) and the [CI/CD pipeline guide](https://mergify.com/learn/cicd-pipeline) cover how the stages fit together. ## How do you get the data without touching your pipeline? Job-level data comes from the CI provider itself, so there is nothing to instrument. Enable [CI Insights](https://mergify.com/product/ci-insights) on your repositories from the GitHub integration page in the dashboard, then re-run a workflow or push a commit. Durations, execution counts, failure rates, and cost show up on the Jobs page on the next run. Jenkins and Buildkite have their own [setup guides](https://docs.mergify.com/ci-insights) and land in the same view. Two controls do most of the work on that page. The date range selector sets the window your percentiles are computed over, and the grouping options let you split the same job by branch, runner group, or label. Grouping is how you catch the job that is fine on `main` and awful on pull requests, or fine on one runner label and awful on another. Each job also carries a health status of healthy, flaky, or broken, based on its recent run history. A flaky job is one that produced two different conclusions on the same commit, which is a fact about your CI rather than an opinion about it. If flakiness is what you are chasing rather than duration, [stopping flaky tests from blocking your merge queue](https://mergify.com/guides/stop-flaky-tests-blocking-merge-queue) picks up from there. ## Which job is actually your bottleneck? Sort by duration, note the p95 rather than the average, then multiply by executions. Say a 30-day window on one repository comes back looking like this: Example 30-day CI job ranking: duration percentiles, execution counts, and total time per job | Job | p50 | p95 | Executions | Total time | How to read it | | --- | --- | --- | --- | --- | --- | | `integration-tests` | 18m | 41m | 60 / day | 18h / day | Your bottleneck. Long, frequent, and a p95 more than double the median, so it is unstable on top of slow. | | `build-docker` | 9m | 11m | 60 / day | 9h / day | Predictable, but it runs on every pull request including the ones that only touch docs. The easiest hours to win back. | | `unit-tests` | 4m | 5m | 60 / day | 4h / day | Healthy. Short, tight spread, and already on the critical path of nothing. Leave it alone. | | `e2e-nightly` | 52m | 58m | 2 / day | 1.7h / day | The number everyone quotes in meetings, and the last one worth speeding up. | The numbers are made up, the shape is not: the job everyone names ends up fourth on the list. Ranked by single-run duration, `e2e-nightly` wins by a mile. Ranked by total time, it costs a tenth of what `integration-tests` does, and speeding it up would change nobody's afternoon. The gap between p50 and p95 is the second signal to read. A tight spread means a job that behaves, and shaving it takes real engineering. A p95 at double the median means something intermittent is going on, usually a shared service that times out or a cache that misses. That one is often cheap to fix once you can see it, which is why it is worth opening the executions view and clicking into the slow runs directly. Each execution has a detail panel with the full job timeline and the test results that came with it, so you can trace a regression back to the commit that introduced it. ## Is the job slow, or is the machine? A job that spends six minutes queued and four minutes running reports ten minutes, and every hour you spend adding caches to it will buy you nothing. The Runners page splits those apart. Configure which runner groups to track first, optionally filtered by label, then expand any runner row to get duration percentiles at median, p95, and p99, queue time broken out separately, total runs, failure rate, and cost. Each runner also gets a health status of healthy, unstable, or degraded, scored against its own group, so a node that is limping shows up next to peers doing the same work rather than in the abstract. High queue times across a whole group point at fleet capacity. One runner underperforming its group points at that box. The [runners documentation](https://docs.mergify.com/ci-insights/runners) covers the fleet configuration, and we have no reason to push you toward any particular provider, since we do not sell runners. Do this before you touch any test code. Roughly half the "slow CI" complaints we see resolve into "the fleet is too small at 10am", and that is a purchasing decision rather than an engineering project. ## How do you stop the job from running at all? The cheapest minute is the one you never spend. Look back at `build-docker`: nine minutes, sixty times a day, on every pull request whether or not the change could possibly affect the image. Making it faster is a week of work. Not running it on documentation changes is an afternoon. Declare the areas of your repository as scopes in `.mergify.yml`: ``` scopes: source: files: web: include: - apps/web/**/* api: include: - services/api/**/* docs: include: - docs/**/* ``` Then have one small job resolve the scopes for the pull request and gate the expensive jobs on the ones they cover: ``` jobs: detect-scopes: runs-on: ubuntu-24.04 outputs: api: ${{ fromJSON(steps.scopes.outputs.scopes).api }} steps: - uses: actions/checkout@v5 - id: scopes uses: Mergifyio/gha-mergify-ci@v25 with: action: scopes integration-tests: needs: detect-scopes if: ${{ needs.detect-scopes.outputs.api == 'true' }} uses: ./.github/workflows/integration-tests.yaml secrets: inherit ``` The `scopes` action inspects the diff and returns a JSON map of scope names set to true or false, which each downstream job reads in its `if` condition. Add a final gate job using the `wait-jobs` action and make that one status required in your branch ruleset, so skipped jobs never look like missing coverage. Full setup is in the [monorepo CI documentation](https://docs.mergify.com/monorepo-ci/github-actions), and the [monorepo CI guide](https://mergify.com/learn/monorepo/ci) covers when this pays off and when it does not. The same scopes then feed the merge queue. When Mergify assembles a batch it prefers pull requests that share scopes, so a batch tends to need one CI run rather than one per change, and a failure inside it is cheaper to bisect. If you do not have a queue yet, start with [setting one up on GitHub](https://mergify.com/guides/setup-github-merge-queue), then come back and wire the scopes in. What to watch once it is running is covered in [queue analytics](https://mergify.com/learn/queue-analytics). ## How do you stop it creeping back? CI gets slow the way weight gets gained, half a minute at a time, and nobody notices until the number is embarrassing. The fleet view carries a week-over-week comparison so you can answer "is CI getting worse" without building a dashboard for it. Pair it with the Slack integration for the job outcomes you care about, and with the PR summary comments that post what broke and where, so a regression gets caught by the person who introduced it instead of by the whole team a month later. Re-run the ranking once a month. It takes ten minutes and the answer changes, because the job you fixed drops off the list and something you have never looked at moves up. ## FAQ ### Do I have to change my CI config to see job durations? No. Job-level data comes from the CI provider itself, so enabling CI Insights on the repository is enough and the Jobs page fills in on the next workflow run. Test-level data is the part that needs a step in your workflow to upload JUnit reports. ### Does this work outside GitHub Actions? Jenkins and Buildkite are supported today and have their own setup guides, with CircleCI coming. Flaky detection and auto-retry behave the same way whichever CI produced the job, which is the point of putting several providers in one view. ### Why sort by p95 instead of the average? The average hides the runs that hurt. A job with a 4-minute median and a 30-minute p95 is a job that blocks somebody badly once or twice a day, and an average of 6 minutes makes it look fine. Sort by p95 to find pain, by median to find steady cost. ### Should I fix the longest job or the most frequent one? Multiply duration by executions and start at the top of that list. A 9-minute job that runs 60 times a day burns five times more machine time than a 52-minute nightly, and it is also the one sitting between your engineers and their merges. ### My CI provider already has a dashboard. What does this add? Health classification per job (healthy, flaky, or broken) rather than a raw list of runs, the same view across every provider you use, queue time split out from execution time, and the ability to act on what you find with auto-retry, quarantine, and scoped runs. ### How much history do I need before the ranking means anything? A couple of weeks. Percentiles need volume to settle, and a shorter window will over-weight whatever incident happened that Tuesday. Once you have a month, compare windows rather than absolute numbers. Last updated: July 29, 2026 ## See where your CI minutes go Enable CI Insights on a repository and get the ranking on your next workflow run. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how CI Insights works](https://mergify.com/product/ci-insights) --- Source: https://mergify.com/guides/manage-dependent-prs-depends-on Guide # How to manage dependent PRs with Depends-On You declare a pull request dependency by adding a `Depends-On:` line to the pull request body. Mergify reads it and refuses to merge that pull request until every one it names has landed. There is nothing to configure. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated July 29, 2026 ## In one paragraph Put `Depends-On: #142` in the body of the pull request that has to wait, one line per dependency. Mergify blocks the merge until #142 is merged, then lets the pull request through the normal path. The reference can point at another repository in your organization, so a schema change and the service that reads it can merge in the right order without anybody watching the clock. ## What goes wrong with dependent pull requests? A migration has to land before the code that queries the new column. An API change in one repository has to be live before the client in another one starts calling it. The ordering is obvious to the author and invisible to everyone else, including the automation. So the ordering gets encoded in people. A `do not merge` label, a comment saying "merge after #142", a message in the release channel. It holds until the person who wrote it is in a meeting and someone else clears the review queue, and then main has code calling a column that does not exist yet. The fix is to write the dependency somewhere the merge automation reads. In Mergify that place is the pull request body. ## How can you say "this one waits for that one"? There are a few ways to express the same ordering, and they do not cost the same thing. | Approach | Scope | What enforces the order | What it costs | | --- | --- | --- | --- | | Depends-On header | Same repository or any repository in your organization | Mergify holds the merge until the referenced pull requests land | Each pull request keeps its own branch and its own review | | A stack from mergify stack push | One repository | The queue moves the whole chain as a unit | Branches are chained, so amending a commit low in the stack rewrites the ones above it | | Basing the pull request on the other one's branch | One repository | GitHub retargets the pull request once its base merges | Without a Depends-On marker the queue still treats the pull requests as unrelated | | A do-not-merge label and a person watching | Anywhere | Only while somebody remembers | The label outlives the reason for it, and someone has to come back and remove it | Pick the header when the pull requests are genuinely separate changes that happen to have an order, which is most of the time, and especially when they live in different repositories. Pick a stack when one change was split into commits that only make sense read in sequence. ## How do you declare the dependency? Add a `Depends-On:` line to the body of the pull request that has to wait. One line per dependency, anywhere in the description: ``` Adds the billing API client to the web app. Depends-On: #142 Depends-On: myorg/billing-service#87 Depends-On: https://github.com/myorg/schemas/pull/9 ``` That pull request now merges only after #142, #87 in `billing-service`, and #9 in `schemas` have all merged. Everything else about it stays the same. Reviewers review, CI runs, and the pull request can sit in the queue. The one step Mergify withholds is the merge, and it reports why on the pull request itself, so the next person to look at it does not have to reconstruct the reason. Three reference formats work, and you can mix them in the same body: - `#123` for a pull request in the same repository. - `org/repo#123` for a pull request in another repository. - `https://github.com/org/repo/pull/123`, which is what you get from copying the address bar. Two limits are worth knowing before you rely on this. Circular chains and self-references are ignored rather than reported, so a pull request that depends on itself simply merges. And a dependency between pull requests targeting different branches is not re-evaluated automatically, so if the blocked pull request stays blocked after its dependency merges, comment `@mergifyio refresh` on it. ## How do you depend on a pull request in another repository? Same header, wider reference. This is the case a stack cannot cover, since a stack is a chain of branches inside one repository, and it is the case teams hit hardest: the schema and its consumer, or the shared library and everything that pins it. Two conditions apply. Every referenced repository must have Mergify installed, and it must belong to the same GitHub organization. Mergify is watching those pull requests through the same installation, which is how it knows when they merge. You can combine the dependency with a date when a change is ready but should not ship yet. Both headers live in the same body, and the pull request waits for both: ``` Depends-On: myorg/billing-service#87 Merge-After: 2026-09-01T09:00:00[Europe/Paris] ``` [Both headers](https://docs.mergify.com/merge-protections/builtin) are built in, and neither needs a line of configuration. The people who need them are rarely the people who own `.mergify.yml`. ## What does the merge queue do with a dependency? A dependent pull request goes through the queue like any other. It is tested against the state the branch will have at merge time, and it is held at the merge step while its dependency is open. When the dependency lands, it stops being held. Turn on auto-merge and the header becomes the only brake you need. This configuration queues anything with an approval and green CI, and the dependent pull requests still wait their turn: ``` merge_protections_settings: auto_merge_conditions: true merge_protections: - name: review-and-ci if: - base = main success_conditions: - "#approved-reviews-by >= 1" - check-success = ci ``` Nothing in it mentions dependencies, and nothing has to. Nobody edits a shared config file to declare that this week's migration goes before this week's query change. The queue does read `Depends-On` for one specific shape. When pull requests are both chained by branch (each one based on the previous one's head) and carry the marker, the queue recognizes a [stack](https://docs.mergify.com/merge-queue/stacks) and moves it as a unit: `@mergifyio queue` on the top pull request enqueues the whole chain, predecessors are validated ahead of successors, and if one fails, the successors still in the queue are dequeued instead of being tested against a dependency that just broke. Chained branches without the marker do not get that treatment. The queue handles them as unrelated pull requests. ## What if you are writing the whole chain yourself? When the dependency exists because you split one change into several reviewable commits, do not write the headers by hand. Any stacking tool does it for you, including GitHub's own `gh stack` and the open-source [Mergify CLI](https://docs.mergify.com/stacks/): `mergify stack push` opens one pull request per commit, bases each on the previous one, and adds the `Depends-On: #NNN` marker between them, which is exactly the pair of signals the queue looks for. ``` mergify stack push ``` For a dependency that points outside the stack, put the `Depends-On:` line in the commit message body. It is copied into the pull request description on push, so it survives the next `stack push` instead of being overwritten. Two limits on the stack side: chains deeper than 20 pull requests fall back to per-pull-request queueing, and a draft anywhere in the chain stops the queue from pulling the stack through. Mark it ready before you queue. ## Where this fits `Depends-On` is one of the built-in controls of the [Mergify merge queue](https://mergify.com/product/merge-queue#freezes-and-dependencies), next to `Merge-After` and the rules you write on labels, files, and authors. GitHub branch protection has no equivalent, which is why the ordering usually ends up as a label and a reminder. It pays off most on top of a queue, since the queue is what tests each pull request against the state it will actually land on. If you do not have one yet, start with [setting one up on GitHub](https://mergify.com/guides/setup-github-merge-queue), and read the [merge queue guide](https://mergify.com/learn/merge-queue) for what the queue does to your branch. For holding a whole branch rather than one pull request, [running a queue during a code freeze](https://mergify.com/guides/merge-queue-code-freeze) covers the other end of the same problem. ## FAQ ### Do I need to configure anything in .mergify.yml to use Depends-On? No. Depends-On is a built-in merge protection. Mergify reads the header from the pull request body and enforces it with no configuration. ### Can a pull request depend on one in a different repository? Yes, as long as both repositories belong to the same GitHub organization and both have Mergify installed. Reference it as org/repo#123 or with its full URL. ### What happens if two pull requests depend on each other? Nothing. Cycles and self-references are ignored, so a circular chain will not deadlock your merges. It also will not warn you, so it is worth reading the chain once before you rely on it. ### Does GitHub have an equivalent? No. Branch protection and rulesets have no notion of one pull request waiting for another, which is why teams end up enforcing it with labels and reminders. ### My dependency merged but the pull request is still blocked. What now? That happens when the two pull requests target different branches, where the evaluation is not automatic. Comment @mergifyio refresh on the blocked pull request and Mergify re-evaluates it. ### Do I still add the header by hand when I use stacks? Not within the stack. mergify stack push writes Depends-On between consecutive pull requests for you. Add the header yourself only for dependencies outside the stack, such as a pull request in another repository, by putting it in the commit message body. Last updated: July 29, 2026 ## Stop merging things in the wrong order Install Mergify, add a Depends-On line to your next dependent pull request, and let the queue handle the ordering. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how the merge queue works](https://mergify.com/product/merge-queue#freezes-and-dependencies) --- Source: https://mergify.com/guides/merge-pull-requests-from-coding-agents Guide # How do I merge pull requests from Claude Code and other coding agents safely? Keep your required checks and a human approval, then let a merge queue do the merging. The agent queues its own pull request, the queue tests it against the real next main, and your rules decide what lands. The agent never merges directly. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 4, 2026 ## In one paragraph Generating the pull request is the part agents already do well. The risk sits in the steps after it, and each has a specific way to go wrong: CI that never starts because of the token the agent used, an approval from another bot, a push that drops the pull request out of the queue, a flaky test the agent tries to fix by rewriting working code. This guide walks those steps in order, with the config and the CLI calls to put in the agent's instructions. ## What happens after the agent opens the pull request? Five things have to go right before the change is on main: | Stage | What should happen | How it goes wrong | | --- | --- | --- | | CI starts | Your required checks run on the branch. | The runs never start, because of the token the agent used. | | Review | A person approves the change. | Another agent's approval satisfies the rule. | | Queueing | The agent asks for the queue. The pull request waits for its approval and checks. | The agent keeps pushing, and each push cancels the request. | | Queue run | CI runs on the latest main plus the pull requests ahead of this one. | It fails, and the agent retries blindly or rewrites working code. | | Merge | The queue merges the state it just tested. | Someone, or something, merges around the queue. | ## Will CI even run on the agent's pull request? It depends on the token. A workflow that opens a pull request with the default `GITHUB_TOKEN` gets [runs in an approval-required state](https://docs.github.com/en/actions/concepts/security/github_token): nothing happens until someone with write access presses the button. GitHub's fix is a GitHub App token or a personal access token. - **Claude Code GitHub Actions:** if you pass `github_token: ${{ secrets.GITHUB_TOKEN }}` to `claude-code-action`, [Anthropic's docs](https://code.claude.com/docs/en/github-actions) say to remove it so the action authenticates as the Claude GitHub App, or to pass a custom app token. - **GitHub Copilot:** workflows [do not run automatically](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/cloud-agent/review-copilot-prs) when Copilot pushes to a pull request. Someone clicks **Approve and run workflows** in the merge box. - **An agent on a laptop or in a sandbox:** it pushes with whatever credential you gave it. A personal token or an app token starts CI normally. A required check that never starts is easy to miss, since the pull request just sits there looking pending. Check this first when an agent's pull requests stop moving. ## Who approves it? A person, and you can write that down. A required approval on GitHub counts any reviewer with write access, which can include a bot account running a review agent. Turn on required review from code owners in the branch's ruleset and name a team of people as the owners: ``` # .github/CODEOWNERS * @acme/platform ``` Only a member of that team can now give the approval that counts, and Mergify reads the ruleset, so the queue waits for it too. GitHub has a similar rule of its own for Copilot: [your approval of a pull request you asked Copilot to open does not count](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/cloud-agent/review-copilot-prs) toward required reviews, so someone else has to approve it. [Testing AI-generated pull requests](https://mergify.com/guides/test-ai-generated-pull-requests) covers what the checks themselves should be. ## How does the pull request get into the queue? The agent asks. It can comment `@mergifyio queue` on the pull request, or run `mergify queue add 4821` with the [Mergify CLI](https://docs.mergify.com/cli/usage). Either one can come before the review: the request stays pending until the pull request is approved and its required checks pass, so the agent can ask once and move on to its next task. `mergify queue add` on a pull request that is already queued exits 6 with `PULL_REQUEST_ALREADY_QUEUED`, which an agent can treat as done. One rule for the agent: [stop pushing](https://docs.mergify.com/commands/queue). A push cancels a pending queue request, because the request vouched for the commits that were there when it was made. Once the pull request is in the queue, a push takes it out. Agents that keep polishing a branch after asking for a merge lose their place every time. ## How do I stop an agent from merging around the queue? Instructions are a request, and a token with write access can still call `gh pr merge`. If the queue is meant to be the only path to main, enforce it on GitHub's side. [Exclusive mode](https://docs.mergify.com/merge-queue/deploy) uses a ruleset with Mergify on the bypass list, set up from **Merge Queue > Deployment** in the dashboard, and after that only Mergify can merge. People lose the green button too, which is what you want once the team has settled into the queue. Until then, hybrid mode lets queue merges and manual ones coexist. ## How do I keep agents from crowding out everyone else? Give the queue more capacity. Forty refactors queued overnight should not leave a hotfix waiting at nine in the morning. [Batching](https://mergify.com/guides/cut-ci-costs-batching-bisection) lets several of them share one CI run, and it takes one setting: ``` queue_rules: - name: default batch_size: 5 ``` In a monorepo, [scopes](https://mergify.com/learn/monorepo/merge-queue) also keep changes in different parts of the repository out of each other's way. ## What does the agent do when something fails? It asks before it changes code. A person who sees a test fail remembers it has been doing that all week. An agent has no such memory, so it rewrites something, reruns, and either burns its run or lands a pointless change. Three CLI calls cover the loop: ``` # Ask to queue pull request 4821. It enters once it is approved and green. mergify queue add 4821 # Where is it? If it left the queue: why, and which checks failed. mergify queue show 4821 # Is this failing test flaky, or did the change break it? mergify tests show 'tests/billing/test_invoice.py::test_rounding' --json ``` `mergify queue show` works after the pull request has left the queue too, and prints the reason (`CHECKS_FAILED`, for one) with the job log URL. `mergify tests show` takes the full test name or a glob, since a bare function name matches nothing. Its `--json` output gives each test a `health_status` of `healthy`, `flaky` or `broken` from [Test Engine](https://mergify.com/product/test-engine). Read that field, since the command exits 0 on a flaky test too. It answers from your default branch, so a test the pull request just added has no history to read yet. A batch failure is the queue's problem to solve: it splits the batch, finds the pull request responsible, and merges the rest, so an agent whose pull request is still queued has nothing to do. ## How does the agent learn all this? Install the Mergify skills. They teach the agent everything above: queueing a pull request, reading why it left the queue, checking whether a failing test is flaky, plus stacked pull requests and the configuration. They live in the [mergify-cli repository](https://github.com/Mergifyio/mergify-cli). ``` # Claude Code claude plugin install mergify # Other agents npx skills add Mergifyio/mergify-cli ``` If your agent does not load skills, or you want the rules written down in the repository, put this in `CLAUDE.md`, `AGENTS.md` or wherever it reads its instructions: ``` ## Landing a pull request - Open the pull request and wait for its checks. Never merge it yourself. - When it is ready for review, ask for a review and run `mergify queue add `. It enters the queue once it is approved and green. - Do not push to the branch after that. A push cancels the request, or takes the pull request out of the queue. If a review asks for changes, push them and queue it again. - If a test fails, run `mergify tests show '' --json` before changing any code. If `health_status` is `flaky`, the failure is probably not yours: say so on the pull request and leave the code alone. - If the pull request leaves the queue, run `mergify queue show `. It prints the reason and the failing checks. Fix the cause, push, and queue it again. ``` ## FAQ ### Does this replace Claude Code GitHub Actions? No, the two sit one after the other. claude-code-action runs Claude Code inside a workflow and can turn an issue into a pull request. Mergify starts where that ends, with the pull request, and handles testing it against main and merging it. ### Does this work with Cursor, Codex and Copilot too? Yes. The queue sees a GitHub pull request, whichever tool opened it. The differences are in how each agent authenticates, which decides whether your CI starts on its own, and in what you can put in its instructions. ### Can an agent merge without a human approval? Only if your branch rules allow it. The queue applies the reviews your ruleset requires, so on a branch that requires an approval, an agent's pull request waits for one like any other. ### What credential does the agent need for the Mergify CLI? An admin application key in MERGIFY\_TOKEN, or a user credential from mergify auth login. A ci application key is refused by mergify tests show. ### How does the agent find out the pull request merged? mergify queue show reports it, for a pull request that is still queued and for one that has left: merged by the queue, dequeued with a reason, or never queued. With --json the same answer is machine-readable. Full reference: [the queue command](https://docs.mergify.com/commands/queue), [deploying your merge queue](https://docs.mergify.com/merge-queue/deploy) and [the CLI](https://docs.mergify.com/cli). Last updated: November 4, 2026 ## Your agents open the pull requests. Let the queue land them. Keep your checks and your reviewers, and test every change against the main it lands on. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/guides/merge-queue-code-freeze Guide # How to run a merge queue during a code freeze You run a merge queue during a code freeze by blocking the merge and leaving everything else alone. Schedule a freeze in Mergify, exclude your hotfix label, and pull requests keep queuing and testing while nothing lands on the frozen branch. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated July 29, 2026 ## In one paragraph Create a scheduled freeze with a start, an end, and a timezone, scope it to the branch you are stabilizing, and exclude the label your team puts on hotfixes. Reviews and CI carry on, and the queue keeps testing pull requests against the state the branch will have at merge time. When you delete the freeze, the backlog drains in order, already tested. Nobody has to re-queue anything. ## What should a code freeze actually block? Most code freezes are enforced by asking people to stop. Someone posts in Slack, someone else merges anyway on Thursday afternoon, and the release engineer finds out on Friday. The freeze that works is the one your tooling enforces, and the only thing it needs to enforce is that code stops reaching the release branch. Everything upstream of the merge can keep running. People open pull requests, reviewers review, CI runs, and the queue tests changes against the branch state they will land on. That work is not what puts your release at risk, and stopping it just means the backlog arrives untested the moment the freeze lifts. So before you configure anything, write down two things: which branch is frozen, and what gets through anyway. In practice the second one is always some version of "a hotfix with an approval". Everything below is how you encode those two answers. ## Freeze, pause, or schedule: which one do you need? Mergify has four ways to stop something from merging, and they are not interchangeable. Picking the wrong one is how you end up cancelling CI you wanted to keep, or freezing a repository when you meant to hold a single pull request. | Tool | What it blocks | CI keeps running | Set up with | Use it for | | --- | --- | --- | --- | --- | | Scheduled freeze | Merging, for every pull request matching the freeze conditions | Yes | Dashboard, or mergify freeze create | A dated freeze: release week, holidays, an audit window | | schedule in merge\_conditions | Merging, outside the window you declare | Yes | .mergify.yml | A recurring window: no merges at night or on weekends | | Queue pause | Merging and queue CI, and running queue checks are cancelled | No | Dashboard, or mergify queue pause | A CI outage, where testing anything is pointless | | Merge-After header | One pull request, until a timestamp you set | Yes | The pull request description | A single change that is ready but must not ship before a date | For a calendar code freeze, the scheduled freeze is the one you want. It is the only one that takes both a start and an end date, so it does not depend on the person who set it up remembering to switch it off. ## How do you schedule the freeze? From the [Mergify dashboard](https://dashboard.mergify.com/), open Merge Protections, then Freezes, and click Schedule a Freeze. You give it a reason, a start, an end, and the conditions that decide which pull requests it covers. The same thing from a terminal, which is what you want if the freeze is part of a release runbook: ``` mergify freeze create \ --reason "Release 4.2 stabilization" \ --timezone "Europe/Paris" \ --start "2026-12-18T18:00:00" \ --end "2027-01-05T09:00:00" \ -c "base=main" \ -e "label=hotfix" ``` The `-c` conditions select what the freeze covers, and `-e` excludes what escapes it. Without `-c` the freeze applies to every pull request in the repository, which is fine for an incident and usually too broad for a release. Leave out `--end` and you get an emergency freeze with no expiry, which is the right shape during an incident. Remember to delete it afterwards, because an open-ended freeze nobody remembers is worse than no freeze at all. `mergify freeze list` shows what is active. ## How do hotfixes get through? A freeze with no exception gets bypassed by hand the first time production breaks, and once someone has merged around the freeze the freeze is over. Give the exception a name instead: the `-e "label=hotfix"` exclusion above means a pull request labelled `hotfix` is not covered by the freeze at all. Excluded does not mean unchecked. The pull request still goes through the queue, still needs its approvals, and is still tested against the current state of the branch before it merges. You have made the emergency path visible and auditable rather than removing it. Route those hotfixes through their own queue so they do not sit behind the frozen backlog. This config gives hotfixes an unrestricted queue and holds everything else to working hours: ``` queue_rules: # Hotfixes merge whenever they are green, freeze or not. - name: hotfix queue_conditions: - label = hotfix merge_conditions: - check-success = ci merge_method: squash # Everything else queues and gets tested all week, # but only lands during working hours. - name: default merge_conditions: - check-success = ci - schedule = Mon-Fri 09:00-17:00[Europe/Paris] batch_size: 5 merge_method: squash ``` The `schedule` condition on the `default` queue is a standing freeze in config form. Pull requests still enter the queue at 2am and still get tested, they just wait at the front until the window opens. It covers recurring windows, so use it for weekends and out-of-hours, and use a scheduled freeze for anything tied to calendar dates. Replace `check-success = ci` with the name your own CI reports before you commit this. The [queue rules reference](https://docs.mergify.com/merge-queue/rules) covers the rest of the keys. ## What happens to the queue while merges are blocked? It keeps working. Pull requests enter the queue, Mergify builds the branch state each one would produce, and your CI runs against it. The only step that does not happen is the merge itself, so the queue accumulates changes that are already known to be good. This is the difference between a freeze and a pause. [Pausing the queue](https://docs.mergify.com/merge-queue/pause) stops the scheduling too and cancels the checks it can cancel, which is what you want during a CI outage and exactly what you do not want during a release freeze. If your CI is healthy and you are just holding the door shut, freeze rather than pause. One pull request that needs to wait for a date, rather than a whole branch, does not need a freeze at all. Add a header to its description and Mergify holds it: ``` Merge-After: 2026-01-05T09:00:00[Europe/Paris] ``` ## How do you lift a freeze without a stampede? Delete the freeze and the queue starts merging again, in the order pull requests joined it: ``` mergify freeze list mergify freeze delete --reason "Release 4.2 shipped" ``` A freeze that lasted two weeks leaves a backlog, and merging it one pull request at a time means one full CI run each. [Batching](https://mergify.com/learn/merge-queue/batching) tests several together and bisects the batch when it fails, which is the difference between draining a 40 pull request backlog in an afternoon or in two days. Raise `batch_size` before you lift the freeze, not after. The other thing worth doing before you unfreeze is checking the queue for changes that have gone stale. Anything opened before the freeze started has had two weeks for the world to move under it. The queue will catch that on its own, since it tests against the current branch state rather than the state the author saw, but a failing batch is easier to read when you were expecting it. ## Where this fits Freezes are one of the controls of the [Mergify merge queue](https://mergify.com/product/merge-queue#freezes-and-dependencies), alongside rules on labels, files, authors, and pull request dependencies. They work on top of the queue itself, which is what makes the "keep testing while blocked" behaviour possible in the first place. If you do not have a queue yet, start with [setting one up on GitHub](https://mergify.com/guides/setup-github-merge-queue), then come back and add the freeze. For the mechanics of what a queue does to your branch, the [merge queue guide](https://mergify.com/learn/merge-queue) is the longer explanation. ## FAQ ### Should people stop opening pull requests during a code freeze? No. That is the mistake most freezes make. Reviews, CI, and queueing can all keep running while merging is blocked. The freeze only needs to stop code from reaching your release branch. ### What is the difference between freezing merges and pausing the queue? A freeze blocks merges while your checks keep running, so the queue stays warm. Pausing stops the queue from scheduling or merging anything and cancels the checks it can cancel. Use a freeze for release and calendar windows, and pause for a CI incident where running tests would only produce noise. ### Can we freeze one branch and leave the others open? Yes. A scheduled freeze takes matching conditions, so a freeze created with base=main leaves every other branch merging normally. The same works for a freeze scoped to a label, an author, or a file path. ### How do hotfixes get through the freeze? Add an exclusion when you create the freeze, for example an exclusion on label=hotfix. Pull requests carrying that label are not covered by the freeze and merge through the queue as usual, tested against the real state of the branch. ### What happens to pull requests that were already in the queue when the freeze started? They stay in the queue in the same order. Mergify keeps testing them and holds them at the front instead of merging. When you lift the freeze, they merge in order without anyone re-queueing them. ### Can we set up a recurring freeze without touching the dashboard? Yes. A schedule condition in merge\_conditions covers recurring windows such as weekends or out-of-hours, and it lives in .mergify.yml alongside the rest of your config. Scheduled freezes cover one-off calendar windows, which a recurring schedule cannot express. Last updated: July 29, 2026 ## Schedule your next freeze Install Mergify, schedule your next freeze, and let the queue keep testing while nothing lands. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how the merge queue works](https://mergify.com/product/merge-queue#freezes-and-dependencies) --- Source: https://mergify.com/guides/merge-queue-if-ci-is-fast Guide # Do I need a merge queue if my CI is fast? Yes, once your team merges more than a few pull requests a day. Fast CI shortens each run, but each run still tests the main the pull request branched from, so two green pull requests can land and break main. A merge queue tests what will actually land. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 4, 2026 ## In one paragraph Faster runners fix how long you wait for CI. They do not change what CI tested. A pull request is checked against main as it was at its last run, and it merges onto main as it is now. The gap between the two is mostly review and waiting time, so cutting a 30-minute pipeline to 2 minutes barely closes it. A merge queue closes it by testing each pull request on the latest main plus everything ahead of it. Fast CI and a queue also multiply: the shorter the run, the more pull requests the queue can clear in an hour. ## What can fast CI not catch? Merge skew, also called a semantic conflict: two changes that are each fine, merge without a conflict, and do not work together. Here is one on a repository with very fast CI: ``` 09:00 main has charge(order) 09:10 PR #41 changes it to charge(order, currency) CI green 09:15 PR #42 adds a refund flow that calls charge(order) CI green 11:20 #41 merges. No conflict: git sees different files. 11:45 #42 merges. No conflict. 11:46 main fails to build: refund calls charge() with a missing argument. ``` Every check was green and git never complained. No run ever saw `#41` and `#42` together, because no run was asked to. A CI that finished in ten seconds would have reported the same two green checks, a little sooner. ## Doesn't faster CI at least shrink the risk? A little. A pull request is exposed from the moment its last CI run started until it merges, and most of that time is spent waiting for a reviewer. These numbers are illustrative, for a pull request that merges three hours after its last run finishes: | CI duration | Review and waiting | Exposed to skew | Reduction | | --- | --- | --- | --- | | 30 minutes | 3 hours | 3 h 30 min | baseline | | 10 minutes | 3 hours | 3 h 10 min | 10% | | 2 minutes | 3 hours | 3 h 02 min | 13% | A 15-fold faster pipeline takes 13% off the window. Your own number is easy to get: compare the merge time of your last twenty pull requests with the time of their last CI run. Whatever merged inside that gap went untested against them. Fast CI is still worth having, for everything else it does. It answers a different question: | Question | Faster CI | Merge queue | | --- | --- | --- | | How long until a developer hears back? | Shorter on every push | No change | | What does each run cost? | Less time per run | Fewer runs, through batching | | Was the pull request tested on the main it lands on? | Only if nothing merged since its last run | Yes | | Can two green pull requests break main? | Yes | No, the second one fails in the queue and never lands | ## Isn't "Require branches to be up to date" enough? On a quiet repository, yes, and it is free. [GitHub's strict status checks](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) refuse the merge until the branch contains the latest main, which closes the gap. Fast CI makes this setting bearable for longer, because each update costs less. The cost is coordination. Every merge makes all the other open pull requests out of date, so someone has to update one, wait for CI, and hope nobody else merges first. With five-minute CI, that is twelve merges an hour at the very best, with a person watching each one. A queue runs the same check and takes over the updating, the ordering and the waiting. ## When do I not need one? Skip it if you merge fewer than about five pull requests a day and main rarely breaks from changes in flight. Small repositories with one team fit this, and a queue would be solving a problem they do not have. Pulling main before you push, or turning on the strict setting above, will carry you. Add one when any of these show up: - Main goes red and every pull request merged that day has a green check. This is the direct signal, and it is worth going through a month of main failures to count them. - People rebase and re-run CI just to get a merge in, or wait for a quiet moment to press the button. - Your merge volume went up without the team growing, which is what coding agents do to a repository. More pull requests in flight means more overlap between them. - A red main stops everyone, so a break costs far more than the one person who has to revert. The [merge queue guide](https://mergify.com/learn/merge-queue) covers how the queue works, and [Mergify vs GitHub merge queue](https://mergify.com/compare/github-merge-queue) covers which one fits. ## What does fast CI do for a merge queue? It sets the speed limit. A queue that tests one batch at a time clears one batch per CI run, so a shorter run is more merges per hour. Teams with fast CI get the most out of a queue: | CI duration | Merges per hour, one at a time | Merges per hour, batch\_size: 5 | | --- | --- | --- | | 30 minutes | 2 | 10 | | 10 minutes | 6 | 30 | | 2 minutes | 30 | 150 | That is the ceiling for one lane with every run passing. Parallel checks raise it further, and a failing batch lowers it while the queue bisects. The setup is one setting: ``` queue_rules: - name: default batch_size: 3 ``` Your required checks and approvals stay in your branch's ruleset on GitHub, and the queue applies them. Queue a pull request by commenting `@mergifyio queue`. Your CI keeps running on your runners. The queue only decides what gets tested, and [Direct Merge](https://docs.mergify.com/merge-queue/direct-merge) skips the run entirely when a pull request already contains the tip of main. [Setting up a merge queue on GitHub](https://mergify.com/guides/setup-github-merge-queue) has the full walkthrough. If your CI is not as fast as you would like, [finding your slowest CI jobs](https://mergify.com/guides/find-slowest-ci-jobs) is the place to start, and [testing AI-generated pull requests](https://mergify.com/guides/test-ai-generated-pull-requests) covers the setup for teams whose volume comes from agents. ## FAQ ### What is merge skew? Merge skew is the gap between what CI tested and what actually lands. A pull request is tested against main as it was at its last run, and it merges onto main as it is now. When something merged in between changes the behavior it relies on, you get a semantic conflict: git reports nothing, both pull requests are green, and main is broken. ### Is a semantic conflict the same as a merge conflict? No. A merge conflict is two changes touching the same lines, and git stops you. A semantic conflict is two changes that merge cleanly and do not work together, such as a function signature changing in one file while a new caller appears in another. Git has no way to see it, only a test run on the combined code does. ### Does GitHub's 'Require branches to be up to date' setting solve this? Yes, for a small repository. Each merge makes every other open pull request out of date, so someone has to update it and wait for CI again before it can merge. A queue does the same safety check and handles the updating and the ordering for you. ### Will a merge queue slow down my fast CI? A pull request waits for its turn and for one more CI run, and on a busy day batching and parallel checks are what keep that wait short. On a quiet day it adds nothing: a pull request that already contains the tip of main merges straight away, with no queue run. ### Do I have to change CI providers or runners to use a queue? No. The queue decides what gets tested and in what order. Your CI, on your runners, does the testing, on a draft pull request the queue opens. A faster runner makes the queue faster, so the two work together. Mergify requires GitHub as the code host. Full reference: [merge queue](https://docs.mergify.com/merge-queue), [pull request lifecycle](https://docs.mergify.com/merge-queue/lifecycle) and [performance tuning](https://docs.mergify.com/merge-queue/performance). Last updated: November 4, 2026 ## Keep the fast CI. Stop breaking main. Add a merge queue in front of main and your CI tests what actually lands. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/guides/setup-github-merge-queue Guide # How to set up a merge queue on GitHub You set up a merge queue on GitHub by installing the Mergify app, committing a small `.mergify.yml`, and letting Mergify test and merge your pull requests against the real state of main. This guide walks through each step with a config you can copy. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated July 10, 2026 ## In one paragraph Install the Mergify GitHub App, add a `.mergify.yml` that defines a queue and queues approved pull requests, then approve a PR. Mergify rebuilds each PR onto the state main will have at merge time, runs your CI against it, and merges only when it passes. Main stays green even when several people ship at once. ## Before you start You need two things in place: - A GitHub repository whose CI reports at least one status check on pull requests. If your pull requests already show a green check, you are ready. - Permission to install a GitHub App on the organization or repository. Installing Mergify takes an admin one click. Mergify works on top of whatever CI you already run, so there is nothing to rewrite before you begin. For the concept behind what the queue does, the [merge queue guide](https://mergify.com/learn/merge-queue) covers the mechanics in depth. ## Step 1: Install the Mergify GitHub App Open the [Mergify dashboard](https://dashboard.mergify.com/) and install the GitHub App on the repository you want to queue. Grant it access to that repository. This is the only permission step, and you can start with a single repository while you evaluate. ## Step 2: Add a .mergify.yml Commit this file as `.mergify.yml` at the root of your repository. It defines one queue and adds approved pull requests to it. ``` queue_rules: - name: default merge_conditions: - check-success=ci batch_size: 5 merge_method: squash pull_request_rules: - name: Queue approved pull requests conditions: - "#approved-reviews-by>=1" - check-success=ci actions: queue: name: default ``` Here is what each part does: | Key | What it does | | --- | --- | | `queue_rules` | Defines your named queues and how the pull requests in them get tested and merged. | | `merge_conditions` | The checks a pull request must pass against the queued state before it lands. Replace check-success=ci with the name of your own CI check. | | `batch_size` | How many pull requests are tested together in a single CI run. Higher batches mean fewer CI runs and a smaller bill. | | `merge_method` | How the pull request lands on main: merge, rebase, squash, or fast-forward. | | `pull_request_rules` | The queue action here adds a pull request to the queue once it is approved and its own CI is green. | Replace `check-success=ci` with the name of your own CI check before you commit. The [queue rules reference](https://docs.mergify.com/merge-queue/rules) lists every key you can set on a queue. ## Step 3: Queue your first pull request Open a pull request and get it approved. The `queue` rule adds it to the queue as soon as it has an approval and its own CI is green. If you would rather add one by hand while you test, comment `@mergifyio queue` on the pull request. ## Step 4: Watch it merge Mergify builds a temporary branch that represents main plus the pull requests ahead in the queue plus this one, runs your CI against it, and merges the pull request when the checks pass. If they fail, the pull request leaves the queue and main is never touched. You can follow all of this live from the dashboard. ## Tuning the queue The config above is enough to run a queue. Two settings pay off quickly once your volume grows. [Batching](https://mergify.com/learn/merge-queue/batching) tests several pull requests in one CI run and bisects on failure, which cuts most of the queue CI bill. [Two-step CI](https://mergify.com/learn/merge-queue/two-step-ci) runs light checks on every pull request and saves the full suite for the queue, so you only pay for the heavy run on pull requests that are about to land. If you are weighing Mergify against the built-in option, [Mergify vs GitHub merge queue](https://mergify.com/compare/github-merge-queue) lays out where the native queue is enough and where it stops being enough. ## FAQ ### Do I need to change my GitHub Actions workflows? No. Mergify pushes each candidate to a temporary queue branch and opens a draft pull request for it, so any workflow you already trigger on push or pull\_request runs there without edits. ### What is the ci in check-success=ci? It is the name of the status check Mergify waits for. Replace ci with the exact name your CI reports (for GitHub Actions this is usually the job name). You can list several checks under merge\_conditions. ### Does this replace GitHub's native merge queue? Yes. Only one queue can own merging into a branch, so you run Mergify or GitHub's native queue per repository, not both. The Mergify vs GitHub merge queue comparison has the side-by-side. ### How do I try it without touching my busy repositories? Install Mergify on one low-traffic repository first, commit the config there, and watch a few pull requests go through before you roll it out more widely. ### Can I make hotfixes jump the queue? Yes. Add a second queue rule with a higher priority, or use a priority rule keyed on a label, so urgent pull requests move to the front. The priority docs cover the syntax. Last updated: July 10, 2026 ## Set up your merge queue Install Mergify, commit the config above, and queue your first pull request. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/guides/stop-flaky-tests-blocking-merge-queue Guide # How to stop flaky tests from blocking your merge queue Stop flaky tests from blocking your merge queue in three moves: make the queue tolerant of transient batch failures with `skip_intermediate_results`, retry jobs that failed for a known-flaky reason, and quarantine the tests that keep lying. This guide sets up all three. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated July 29, 2026 ## In one paragraph Commit the `.mergify.yml` below so a failing batch that a larger passing batch contradicts stops dequeuing good pull requests. Add an Auto-Retry rule so jobs that fail on known flakiness re-run themselves. Then upload your test reports and quarantine your worst offenders, so a test that fails half the time on the same commit records the failure without blocking the merge. ## Why does one flaky test stall the whole queue? Outside a queue, a flaky test costs you one rerun. Inside one, the cost compounds. The queue tests pull requests in batches, so when a batch goes red, Mergify assumes the batch is guilty: it cancels the batches behind it, puts those pull requests back in the queue, and splits the failed batch to find the culprit. That is exactly the right behavior when a real bug is hiding in there, and exactly the wrong tax to pay for a test that flipped a coin. Meanwhile the author of the pull request at the front sees a failure on code that did nothing wrong and starts debugging it. That hour is the real cost, and it repeats every time the test misfires. One setting will not cover it. Flakiness shows up at three altitudes (a batch, a job, a single test) and each one has its own answer. For what makes tests flaky in the first place, and framework-specific fixes, see the [flaky tests guide](https://mergify.com/learn/flaky-tests). ## Which layer handles which failure? | Layer | What it handles | Where you set it | What happens | | --- | --- | --- | --- | | Queue anti-flake | A batch that fails once, then passes in a larger batch containing the same commits. | `.mergify.yml` | The pull request stays in the queue and merges. Nobody gets paged. | | Auto-Retry | A whole job that failed on a test Mergify already flagged as flaky, or on a transient error you can match in the logs. | `Dashboard, CI Insights → Auto-Retry` | The workflow re-runs the failed jobs before a human ever looks at the red check. | | Quarantine | One test that keeps producing different results on the same commit. | `Dashboard, or mergify tests quarantines add` | The test still runs, but its failure no longer fails the check or blocks the merge. | Set them up in that order. The queue setting takes two minutes and pays off immediately, Auto-Retry needs a rule you tune once, and quarantine needs data before it can tell you what to quarantine. ## How do you stop a flaky batch failure from dequeuing good pull requests? Turn on `skip_intermediate_results`. With it, Mergify stops requiring every intermediate batch to pass: if a later, larger batch containing the same changes goes green, the earlier red batch is treated as transient and the pull request still merges. ``` merge_queue: skip_intermediate_results: true max_parallel_checks: 3 queue_rules: - name: default merge_conditions: - check-success=ci batch_size: 5 batch_max_failure_resolution_attempts: 3 pull_request_rules: - name: Queue approved pull requests conditions: - "#approved-reviews-by>=1" - check-success=ci actions: queue: name: default ``` Replace `check-success=ci` with the name of your own CI check before you commit this. This is safe because a genuine bug also fails the larger batch that carries the same code, and that failure still ejects the pull request. Only the contradicted failures get skipped. Two caveats: it relies on the strict ordering of serial mode, so it is unavailable in [parallel mode](https://docs.mergify.com/merge-queue/queue-modes), and you may want it off while you are debugging CI and want to see every failure. The other two keys shape how fast the queue recovers when a batch does fail. `max_parallel_checks` sets how many checks the queue runs at once, and with it how many pieces a failed batch gets split into, so a bisection that would take four sequential rounds finishes in one or two. `batch_max_failure_resolution_attempts` caps how deep that splitting goes before Mergify gives up and ejects the batch. If you are new to batches, the [batching explainer](https://mergify.com/learn/merge-queue/batching) covers the mechanics, and the [batch documentation](https://docs.mergify.com/merge-queue/batches) has the full split algorithm. Do not have a queue yet? Start with [how to set up a merge queue on GitHub](https://mergify.com/guides/setup-github-merge-queue), then come back and add these keys. ## How do you retry a job that failed for a known-flaky reason? Every CI has a rerun button. It works fine for one engineer hitting refresh and not at all for a queue processing hundreds of pull requests a day. [CI Insights](https://mergify.com/product/ci-insights) Auto-Retry does it on rules instead of on attention: go to CI Insights → Auto-Retry in the dashboard, pick your repositories, and add a rule. A rule matches on the attributes of the failed job. The ones that matter here: - **Flaky** set to true, so you only retry jobs Mergify has already classified as flaky. This is the conservative rule to start with, and the one worth having in every repository. - **Log pattern**, a regular expression matched against job logs and check-run annotations. Something like `timeout|connection refused` catches the infrastructure failures that have nothing to do with your code. - **Conclusion**, to separate a job that failed from one that timed out, plus job name, pipeline, branch, and repository filters when you want to keep the rule narrow. Then set the number of retries and the mode: `job` re-runs the failed jobs, `pipeline` re-runs the whole workflow. Because of a GitHub API limitation the retry is triggered at the workflow level either way, but jobs that already passed are preserved rather than re-executed. Full options are in the [Auto-Retry documentation](https://docs.mergify.com/ci-insights/auto-retry). Keep the conditions tight. A blind retry-everything rule hides real bugs that only fail occasionally. A rule that only retries failures you already flagged as flaky keeps the opposite property, which is the one you actually want: a failure that survives the retry means something. ## How do you quarantine a test so its failure stops blocking merges? Some tests are past retrying. A test that passes and fails on the same commit is not reporting on your code any more, and every merge it blocks is a merge blocked for no reason. [Test Engine](https://mergify.com/product/test-engine) quarantines it: the test keeps running and its results keep being collected, but its failure no longer fails the check. First, send your test results to Mergify. Create an application key with the `ci` scope, store it as `MERGIFY_TOKEN`, and wire the upload into the job that already runs your tests. The two details that matter for quarantine are `continue-on-error` on the test step, so the upload step decides the conclusion once quarantined tests are discounted, and `test_step_outcome`, so a crashed runner that produced an empty report is not read as a pass. ``` - name: Run tests id: tests continue-on-error: true run: pytest --junit-xml=test-results.xml - name: Upload test results if: success() || failure() uses: Mergifyio/gha-mergify-ci@v25 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: test-results.xml test_step_outcome: ${{ steps.tests.outcome }} ``` Give it a couple of weeks of runs. The Detection page then lists your unhealthy tests, split into flaky (inconsistent on the same commit) and broken (failing consistently), each with a confidence level and an impact score counting the failed executions it caused. Sort by impact, take the handful at the top, and quarantine them from the dashboard or from the terminal: ``` mergify tests quarantines add -r owner/repo \ --reason "flaky, tracked in MRGFY-1234" \ "tests/auth/test_login.py::test_login_timeout" ``` The CLI form is handy in scripts, and it is the one to hand to an AI coding agent that just hit a flaky test and needs to get out of its own way. Once you trust the classification, turn on auto-quarantine per repository from the Mitigation page and Mergify quarantines newly detected flaky tests for you, with an extra option for known-broken ones. The [quarantine documentation](https://docs.mergify.com/test-engine/quarantine) covers the per-framework setup. ## How do you know it worked? Watch the number of batch splits per day in the queue. That is the metric flakiness moves first, and the one that translates directly into CI minutes: fewer splits means the queue stopped chasing ghosts, and pull requests spend their time waiting on one CI run instead of four. Then go back to the Mitigation page every couple of weeks and look at the quarantine list. Tests whose health improved can come out. Tests that have been quarantined for six months without anyone touching them are telling you something else, and Detection keeps them ranked by impact so you know which one to fix first. Quarantine buys time to fix the test properly. It is not where the test retires. ## FAQ ### Is quarantining a flaky test just ignoring the problem? A flaky test is already lying to you: it says FAIL when the code is fine. Quarantine moves it from gating production to a to-fix list, and it keeps running so you still collect data. What makes it honest is that it is per test, never a global ignore-failures switch, and Detection keeps ranking it by how much CI time it wastes. ### Does skip\_intermediate\_results let a real bug through? No. A real bug fails the larger batch that contains the same code, and that batch failure still removes the pull request from the queue. Only a failure that a passing superset batch contradicts is treated as transient. Turn it off if you need every intermediate state validated for compliance. ### Does it work with parallel queue mode? skip\_intermediate\_results needs the strict cumulative ordering of serial mode, so it is not available in parallel mode. Auto-Retry and quarantine work in both, since they run at the CI level rather than in the queue. ### Do I have to change my test code? No. You add an upload step to the workflow that already runs your tests, and Mergify reads the JUnit reports it produces. Test Engine supports pytest, Jest, Vitest, RSpec, Go, Rust, Playwright, Cypress, and a dozen more. ### How many tests should I quarantine to start? Let detection collect a couple of weeks of data, sort by impact, and quarantine the handful at the top. Those few usually account for most of the queue disruption. Enable auto-quarantine once you trust the classification. Last updated: July 29, 2026 ## Keep the queue moving Commit the config, upload your test results, and let Mergify tell the difference between a flaky test and a broken one. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how Test Engine works](https://mergify.com/product/test-engine) --- Source: https://mergify.com/guides/test-ai-generated-pull-requests Guide # How do I test AI-generated pull requests before they merge? Run your normal required checks on the pull request, then send it through a merge queue so it is tested against the main it will actually land on. Batching keeps that affordable at high volume, and Prevention flags new flaky tests before they merge. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 4, 2026 ## In one paragraph An AI-generated pull request fails the same way a human one does, so it gets the same required checks. What changes is how many are open at once. Each one was tested against a main that has moved by the time it merges, and two green changes can break main together. A merge queue closes that gap: it builds each pull request on the latest main and everything ahead of it, runs your CI on that exact state, and merges only if it passes. Batching stops this from costing one full pipeline per pull request, and Prevention reruns new tests so the flaky ones get flagged while the pull request is still open. ## What changes when an agent wrote the pull request? The traffic. A type error or a failing test looks the same whoever typed the code, so the checks can stay as they are. But an agent can open a dozen pull requests in the time a person opens one, work overnight, and touch the same files as another agent running next to it. Every one of those pull requests is tested against the main it branched from. With that many in flight, main moves between the test and the merge far more often, and the classic break gets common: [two changes that each pass on their own and fail once both are on main](https://mergify.com/guides/merge-queue-if-ci-is-fast). A review checklist will not catch this, because neither diff is wrong. So there are two questions to answer before a merge. Does this change work, and does it still work on the main it is about to land on? Most advice on testing AI-generated code stops at the first one. ## What does each check catch? Each layer covers something the one before it cannot see, so none of them replaces another: | Check | Runs | Catches | Misses | | --- | --- | --- | --- | | Lint, types, unit tests | Every push to the pull request | Mistakes inside the change itself | Anything that depends on what else merged | | Human review | Before the pull request can enter the queue | The wrong fix, the wrong design, a test that asserts nothing | A clash with a change that merges after the review | | Test reruns (Prevention) | While the pull request is open | A new test that passes and fails on the same commit | A test that is consistent and wrong | | Merge queue run | Just before the merge | A clash with the latest main and the pull requests ahead of it | A bug your test suite does not cover | The last row is the limit of the whole setup. If an agent writes a test that asserts nothing, every layer but review will wave it through, which is why the human approval stays in front of the queue. ## How do I test a pull request against the main it lands on? Put a [merge queue](https://mergify.com/learn/merge-queue) between the approval and main. When a pull request reaches its turn, the queue builds a temporary branch holding the latest main, the pull requests ahead of it and this one, opens a draft pull request so your CI runs on that branch, and merges only if it passes. If it fails, the pull request leaves the queue with the reason, and main never sees it. The approval and the required checks stay where they already are, in your branch's ruleset on GitHub. Mergify reads them, so a pull request enters the queue only once a person has approved it and its checks are green. The queue itself needs one setting: ``` queue_rules: - name: default batch_size: 5 ``` `batch_size` lets five queued pull requests share one CI run. The next section prices it. To queue a pull request, comment `@mergifyio queue` on it. If you have no queue yet, [setting up a merge queue on GitHub](https://mergify.com/guides/setup-github-merge-queue) takes a few minutes. ## How do I keep CI affordable at this volume? Test several pull requests in one run. Forty pull requests a day on a 30-minute pipeline is twenty runner-hours a day if the queue tests them one at a time. With `batch_size: 5` that becomes eight runs and four hours, plus a few extra runs on the days a batch fails and the queue bisects it to find the culprit. The rest of the batch still merges. Two more settings take runs off the bill. A pull request that already contains the tip of the base branch merges with no queue run at all, which [Direct Merge](https://docs.mergify.com/merge-queue/direct-merge) does without any configuration. In a monorepo, [scopes](https://mergify.com/learn/monorepo/merge-queue) keep a frontend change from waiting behind a database migration, which matters once agents work across the whole repository at the same time. [Cutting CI costs with batching and bisection](https://mergify.com/guides/cut-ci-costs-batching-bisection) has the math for picking a batch size. ## How do I catch flaky tests before they land? A new test that passed once has proved very little, and a repository with agents in it gains a lot of new tests. [Prevention](https://docs.mergify.com/test-engine/prevention), part of [Test Engine](https://mergify.com/product/test-engine), reruns the tests a pull request adds on the same commit, within a CI time budget you set. A test that gives two different results is flagged as flaky, with its health status and a confidence level next to the pull request, so the reviewer can send it back before it reaches main. To turn it on, install the [Mergify plugin for your test framework](https://docs.mergify.com/test-engine/test-frameworks) and set a Prevention Budget on the Test Engine configuration page. Tests that are already flaky on main are a different job, and [detecting and quarantining them](https://mergify.com/guides/detect-and-quarantine-flaky-tests) keeps them from knocking good pull requests out of the queue. ## What should an agent do with a red check? Ask whether the test is flaky before rewriting anything. A red test looks the same whether the change broke it or it fails one run in twenty, and an agent that guesses wrong spends its run rewriting code that was fine. The Mergify CLI answers from the test's history: ``` mergify tests show 'tests/billing/test_invoice.py::test_rounding' --json \ | jq '.tests[] | {test_name, health_status}' ``` `health_status` is `healthy`, `flaky` or `broken`. Read the JSON rather than the exit code, which only reports whether the lookup worked. The command needs an `admin` application key or a user credential from `mergify auth login`. [Merging pull requests from coding agents](https://mergify.com/guides/merge-pull-requests-from-coding-agents) covers the rest of the loop, including what to put in the agent's instructions. ## FAQ ### Do AI-generated pull requests need different CI checks from human ones? No. Keep one set of required checks. A separate, softer path for agents gives them a way around the strict one, and it leaves you with two pipelines to maintain. What changes is the number of pull requests in flight, which is why the merge step needs a queue. ### Does a merge queue replace code review? No. Review answers whether this is the right change. The queue answers whether the change still works on the main it is about to land on. Require an approval on the branch and you get both, in that order. ### Can an agent's approval count as the required review? Only if you let it. A required approval on GitHub counts any reviewer with write access, and an agent running under its own account can have that. To require a person, turn on required review from code owners and list a team of people in CODEOWNERS. ### Will a merge queue slow the agents down? The agent is not waiting on it. It opens the pull request, it gets queued, and the agent moves on to the next task. Batching is what keeps the queue itself from falling behind when pull requests arrive faster than one CI run per merge can clear them. ### Does this work with my CI? Mergify works on GitHub, with any CI that reports its results there: GitHub Actions, Buildkite, Jenkins, CircleCI and others. The queue does not run your jobs. It opens a draft pull request on a temporary branch, and your CI runs on it the way it runs on any pull request. Full reference: [merge queue](https://docs.mergify.com/merge-queue), [batches](https://docs.mergify.com/merge-queue/batches) and [Prevention](https://docs.mergify.com/test-engine/prevention). Last updated: November 4, 2026 ## Test every pull request against the main it lands on. Put a merge queue in front of main, whoever wrote the code. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how Merge Queue works](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/learn Learn # From the people who fix this for a living. Long-form guides on the things that actually slow your team down. Written by Mergify engineers, drawn from thousands of CI pipelines and merge queues we have watched run in anger. ## Topics [Guide ### Merge Queue Why your main branch keeps breaking, and what a modern merge queue actually does to keep it green. Read the guide →](https://mergify.com/learn/merge-queue)[Guide ### Continuous Integration Every change, built and tested on every push. The mechanics are well known. What separates teams that get value from CI from teams that just pay for it is what they do with the signal. Read the guide →](https://mergify.com/learn/continuous-integration)[Guide ### CI/CD Pipeline Build, lint, test, package, deploy. The order is not arbitrary. Each stage filters the work so the slow, expensive ones only run on changes that already cleared the fast checks. Read the guide →](https://mergify.com/learn/cicd-pipeline)[Guide ### Trunk-Based Development One branch, short-lived feature work, a main that is always shippable. The model is simple. The plumbing that makes it work in a real org is what teams skip and regret. Read the guide →](https://mergify.com/learn/trunk-based-development)[Guide ### Flaky tests Flaky tests are not random noise. They follow patterns, and the patterns are framework-specific. Named, fixed, quarantined. Read the guide →](https://mergify.com/learn/flaky-tests)[Guide ### Playwright Microsoft's end-to-end testing framework. Real browser, auto-waiting actionability, multi-language support. The default for new browser-test suites and what makes a Playwright flake different from a Cypress flake. Read the guide →](https://mergify.com/learn/playwright)[Guide ### pytest Python's de facto testing framework. Plain functions, plain asserts, fixtures that compose, a plugin ecosystem that covers everything from async to coverage. How to keep a pytest suite fast and trustworthy past a few thousand tests. Read the guide →](https://mergify.com/learn/pytest)[Guide ### Queue Analytics What to measure on a merge queue: throughput, wait time, batch success rate, failure rate, stage breakdown. The numbers that turn a queue from a black box into a tunable system. Read the guide →](https://mergify.com/learn/queue-analytics)[Guide ### Test Quarantine Stop flaky tests from blocking merges without skipping or deleting them. What to quarantine, how long to leave it there, and the anti-patterns that turn quarantine into a graveyard. Read the guide →](https://mergify.com/learn/test-quarantine)[Guide ### CI Auto-Retry Auto-retry absorbs transient CI noise so engineers do not chase it. The trade is that blanket retry hides real flake patterns. How to retry conditionally and track what auto-retry is hiding. Read the guide →](https://mergify.com/learn/auto-retry)[Guide ### Monorepo One repository for many projects. The benefits are real, and so is the tax on CI and merge coordination. What a monorepo is, why teams adopt one, and how to keep it fast as it grows. Read the guide →](https://mergify.com/learn/monorepo) More guides coming. Want one on a specific topic? [Tell us](https://mergify.com/discovery) and we will probably write it. --- Source: https://mergify.com/learn/auto-retry Guide # CI Auto-Retry Auto-retry exists because CI is noisy. Network blips, runner pool churn, a test that hits a race condition once in a hundred runs. Retrying absorbs the noise so engineers do not chase it. Done well, it is a productivity multiplier. Done bluntly, it is the reason your suite has been quietly broken for six months. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated October 29, 2026 ## In one paragraph Auto-retry re-runs a failed CI job, step, or test so transient noise does not look like a real failure. The trade is that it can also mask a real flake or a slow regression. The fix is to retry conditionally (on known-transient signals), bound the retry budget (one or two attempts), and track the retry rate as a metric, not as exhaust. ## Where auto-retry lives Retries can happen at three levels, each catching a different class of failure: - **Test level:** inside the test framework. A flaky test that hits a race condition gets re-run by pytest-retry, jest.retryTimes, or RSpec's retry gem. Useful when the flake is in the test code itself. - **Job level:** the CI runner re-executes the whole job after a failure. Useful when the failure was the runner dying mid-suite, an image-pull failure, or a network timeout in setup. - **Workflow level:** the higher orchestration layer reruns a whole pipeline. Useful when something failed before tests even started (clone failed, secret rotation glitch, registry hiccup). The three layers protect against different failure modes. Framework retries cannot rerun a job that died. Job retries cannot rerun a single failing test inside an otherwise-passing job. Use whichever level matches the failure. ## Auto-retry on GitHub Actions GitHub Actions has no automatic retry. Its only built-in option is "Re-run failed jobs", a manual button (or `gh run rerun --failed`) that runs every step of the failed jobs again on the same commit. The popular `nick-fields/retry` action retries a step automatically, which is the right tool for a flaky install or network call and an expensive one for tests: it reruns the whole test command. For a flaky test, retry at the test level inside the job, with your framework's own retry, and only on tests you know are flaky. The step-by-step version, with workflows that ran in our sandbox, is in [how to retry failed tests on GitHub Actions](https://mergify.com/learn/github-actions/retry-failed-tests). ## Conditional retry beats blanket retry The biggest lever in any auto-retry policy is conditionality. A blanket retry-on-any-failure rule reruns real failures along with the transient ones, and a test that was about to fail for a real reason gets a second chance to pass on luck. The right rule is narrower: - Retry when the exit code matches a known-transient set (network errors, runner errors). - Retry when the log contains a specific known-flaky pattern ("image pull backoff", "context deadline exceeded"). - Retry tests explicitly marked as flaky in a quarantine list, not the whole suite. - Cap the retry budget to one or two attempts per job. Mergify CI Insights ships rule-based auto-retry that fits this shape: conditions written in the same YAML rule engine as the rest of the queue, full event log so the retry history is debuggable, and per-job retry stats so the policy can be tuned over time. ## Track retries as a metric The single most important thing you can do with auto-retry is measure it. Retry rate is a leading indicator of suite health. If 2% of jobs are passing on retry, the suite is fine. If 30% of jobs are passing on retry, the suite is broken and the retry is hiding it. Watch three numbers in particular: - Percent of jobs that needed a retry to pass. - Top tests by retry count, week over week. - Retries per merged PR (and the trend). When a specific test shows up at the top of the retry list, it is no longer a transient issue. It is a flaky test, and it belongs in [quarantine](https://mergify.com/learn/test-quarantine) or in [the fix pile](https://mergify.com/learn/flaky-tests). ## FAQ ### What is CI auto-retry? Auto-retry is the practice of automatically re-running a failed CI job or test without engineer intervention. The retry can be at the job level (the whole pipeline runs again), the step level (a specific stage), or the test level (a single test reruns). The goal is to absorb transient failures (network blips, runner pool issues, flaky tests) so engineers spend their time on real bugs. ### Is auto-retry the same as test retry? Test retry is one form of it. A failing test gets re-run, often up to two or three times, before being marked as failing for real. Job retry is the broader version: the whole CI run is re-executed when a transient infrastructure error caused the failure. Both are auto-retry. They solve different parts of the same problem. ### What is the risk of auto-retry? It masks signal. A test that flakes 20% of the time will pass on a 3-retry policy almost all of the time, which makes it look fine even though it is still flaky. The result is a slow accumulation of unreliable tests that you only notice once a retry budget runs out or a real regression sneaks through. The mitigation is to track retry rates and treat them as a metric, not as exhaust. ### How many retries are safe? Most teams settle on one or two retries for test-level flakes and one retry for job-level infrastructure errors. Past that, the retry is no longer absorbing transient noise. It is hiding a problem. ### Should I retry on every failure or only specific ones? Only specific ones. A blanket retry policy retries real failures along with the transient ones. The right pattern is to retry on signals you can identify: a specific exit code, a known error message (network timeout, image pull failed, runner lost), or a known-flaky test marked as such. Mergify CI Insights uses rule-based retry so the retry policy can be conditional rather than blanket. ### How is auto-retry different from a retry plugin in my test framework? Framework-level retry (pytest-retry, jest.retryTimes, RSpec's retry gem) runs inside the test process. CI-level auto-retry runs above it: it can re-run the entire job after a runner died, or after a Docker pull failed, neither of which the test framework can see. Both layers are useful, and they protect against different failure modes. ### Can auto-retry slow down CI? Yes, when retries replace fixes. A suite that depends on retry to be green costs 1.2 to 1.5 times its base runtime in steady state and is more variable in latency. Track retry rate over time; if it climbs, the suite is degrading. ## Retry the noise. See the signal. Mergify CI Insights ships rule-based auto-retry, full event log, and per-job retry analytics, so retries become a managed policy instead of a quiet failure-masking habit. [See Mergify CI Insights](https://mergify.com/product/ci-insights)[Read the flaky tests guide](https://mergify.com/learn/flaky-tests) --- Source: https://mergify.com/learn/cicd-pipeline Guide # CI/CD Pipeline A CI/CD pipeline is the automated sequence of steps that turns a code change into running software. Build, run tests, package an artifact, deploy. Every team that ships software has one, whether it is a fifty-line GitHub Actions YAML or a multi-stage Jenkins setup that nobody fully understands. The shape is always roughly the same. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated May 19, 2026 ## In one paragraph A pipeline runs automatically on every code change. It builds the code, runs static checks and tests, produces an artifact (container image, binary, deploy bundle), and ships it through environments toward production. The CI half gives you confidence the change is safe. The CD half gives you the artifact, in a known good state, ready to deploy. The pipeline is defined as code in a YAML file inside the repository, which means changing the pipeline is a normal pull request like any other. ## The canonical pipeline flowchart LR Push\["Push / PR"\] Build\["Build"\] Lint\["Lint & static checks"\] Unit\["Unit tests"\] E2E\["Integration
+ E2E tests"\] Package\["Package /
artifact"\] Deploy\["Deploy to
staging"\] Promote\["Promote to
production"\] Push --> Build --> Lint --> Unit --> E2E --> Package --> Deploy --> Promote style Push fill:#E6F0FF,stroke:#43A7E5,color:#1A1D24 style Promote fill:#E6F8F2,stroke:#1CB893,color:#1A1D24 The standard shape of a CI/CD pipeline. Most pipelines are subsets of this, not supersets. The order is not arbitrary. Each stage filters the work so the slow, expensive ones only run on changes that already cleared the fast checks. ### Build Compile the code. Resolve dependencies. Produce the binary, bundle, or container image. The build step is the gate that says "this change is at least syntactically real." A failure here usually means a typo or a missing import, and you want to find that out before any test ever runs. ### Static checks Linters, type checkers, formatters. These are fast (seconds, not minutes) and catch a real class of bugs that tests do not: unused imports, missing nullability handling, code style drift. They belong second because they cost almost nothing to run. ### Unit tests Fast tests with no I/O. Pure functions, business logic, data transformations. The unit suite should run in a couple of minutes for a healthy project. When it gets slower, the suite has usually drifted into integration territory and needs to be split. ### Integration and end-to-end tests Tests that touch a real database, a real browser, real external services. These are the slow, expensive part of the pipeline. [Playwright](https://mergify.com/learn/playwright) tests live here. So do tests that hit a containerized database, an HTTP test server, or a docker-compose stack of the whole app. This stage is where time and money are usually spent, and where the question of _when_ to run it becomes interesting. ### Package Build the artifact that will actually be deployed. For most modern apps, this is a container image pushed to a registry, tagged with the commit SHA. The package step happens after tests, on the same commit, so the image you ship is provably the image you tested. ### Deploy to staging Push the artifact to a staging environment that looks like production. Run smoke tests against the running deployment. This is the last check before production, and the first chance to catch deployment-level bugs (config errors, secrets misconfigured, database migrations that take longer than the deploy budget). ### Promote to production The same artifact moves from staging to production. In continuous delivery this is a manual click. In continuous deployment it is automatic on green. The choice between the two comes down to the team's tolerance for unattended pushes, not to any technical limitation in the pipeline. ## The two stages teams should keep separate Most pipelines that work in practice have two distinct shapes for two distinct moments: the per-push check and the pre-merge check. | Stage | Runs when | What it does | Target time | | --- | --- | --- | --- | | Fast PR check | Every push to a PR | Build, lint, type-check, unit tests | Under 10 minutes | | Heavy merge check | When the PR is queued to merge | Integration tests, E2E, security scans | Under 30 minutes | | Deploy stage | After merge to main | Package, deploy to staging, smoke test, promote | Under 15 minutes per environment | This is the basic shape of [two-step CI](https://mergify.com/learn/merge-queue/two-step-ci). Engineers iterating on a PR get fast feedback. The expensive checks run once per merge, on a commit that is already trusted to be reviewable. The team pays for the slow stage on every merge, not on every push, and that is usually the difference between a CI bill that is reasonable and one that gets a meeting. ## Where CI/CD pipelines quietly stop working flowchart LR Build\["Build ✓"\] Lint\["Lint ✓"\] Unit\["Unit ✓"\] E2E\["E2E ✗"\] Block\["Block merge"\] Notify\["Notify author"\] Build --> Lint --> Unit --> E2E E2E --> Block E2E --> Notify style E2E fill:#FDECEA,stroke:#E53935,color:#1A1D24 style Block fill:#FDECEA,stroke:#E53935,color:#1A1D24 A failing pipeline blocks the merge. The harder question is what happens when the failure is not a real one. Pipelines fail in obvious ways (test crashes, build errors) and in less obvious ways that compound over months. ### Slow stages no one fixes A pipeline that runs 35 minutes is not necessarily slower than the same pipeline a year ago. Stages add tests, tests add fixtures, fixtures add I/O. Each individual change is small. The aggregate effect, over a year of normal product work, is that PR feedback time doubles. Without a metric on pipeline duration, the slowdown is invisible until somebody notices that "CI takes forever" has become a permanent state. ### Flaky tests that mask real failures A test that fails on one run and passes on the next teaches engineers to click "rerun." Once that habit forms, a real failure goes through the rerun reflex first, and the pipeline's signal degrades into a question. The fix is to detect flakes systematically and quarantine them out of the required check until they are fixed. See the [flaky tests guide](https://mergify.com/learn/flaky-tests) for the patterns. ### Pipeline drift CI configurations age badly. Versions pin, then drift, then break when something upstream changes. Security patches lag. Cache invalidation logic falls behind the build graph. None of this shows up as a failing pipeline today, but it shows up as a 90-minute incident the week before launch. Treating the pipeline file as production code (review, test changes in branches, monitor) is what keeps this in check. ### Two green PRs that break main together The pipeline runs on each PR independently. If PR #1 renames a function and PR #2 adds a caller, both pass CI in isolation. When they land back to back, main is red. The pipeline cannot catch this by design; the fix is a [merge queue](https://mergify.com/learn/merge-queue) that tests each PR against the actual future state of main before merging. ## The pipeline tools landscape The choice of pipeline tool is settled by the choice of forge in most modern setups. - **GitHub Actions**. The default for GitHub repositories. YAML-based, large marketplace of community actions, fast PR feedback. - **GitLab CI**. Native to GitLab. Integrated with merge requests, container registry, and the rest of GitLab's surface. - **CircleCI**. Independent CI provider. Strong macOS support, fast on the right configuration, used heavily by mobile teams. - **Buildkite**. Hosted control plane, self-hosted runners. The pattern of choice when CI needs to run on the team's own hardware. - **Jenkins**. Long-standing, plugin-rich, still pervasive in enterprise. Powerful, operationally heavy. - **Bazel-driven pipelines**. Inside large monorepos, the CI tool is often a thin shell around a build tool that does selective testing and remote caching. The CI tool runs Bazel; Bazel decides what runs. ## FAQ ### What is a CI/CD pipeline? A CI/CD pipeline is the automated sequence of steps that runs on every code change, taking the change from a pushed commit through build, test, package, and deploy. The continuous integration half ensures the change builds cleanly and tests pass. The continuous delivery (or deployment) half ensures the resulting artifact reaches staging and production. ### What are the stages of a CI/CD pipeline? The common stages, in order, are: clone the repository, install dependencies, build, run static checks (linters, type checks), run unit tests, run integration and end-to-end tests, package the artifact (container image, binary, deploy bundle), deploy to staging, and promote to production. Most pipelines have a subset of these, not all. ### What is the difference between CI and CD? Continuous integration is about the code: every change is built and tested on a shared branch. Continuous delivery is about the artifact: the build output is always in a state that could be deployed to production. Continuous deployment goes further and actually ships every green commit. Most pipelines marketed as CI/CD cover the first two and leave the final promote-to-production step as a click. ### What is a CI/CD pipeline example? On GitHub, a typical pipeline is defined in .github/workflows/ci.yml: a workflow that runs on every push, with jobs for lint, test, build, and deploy. On GitLab, the same shape lives in .gitlab-ci.yml. On CircleCI, .circleci/config.yml. The structure is the same across tools, the syntax differs. ### Why are CI/CD pipelines important? Because they replace human checklist work with automated guarantees. Without a pipeline, somebody has to remember to run the tests, build the image, push it to the registry, and deploy it. Each of those steps gets skipped under pressure. A pipeline removes the option of skipping by making the automation the only path to production. ### How long should a CI/CD pipeline take? For the parts an engineer waits on (PR checks), the practical ceiling is 10 to 15 minutes. Past that, engineers context-switch and the pipeline becomes a tax on productivity. Heavy stages (E2E suites, integration tests, security scans) belong in a second stage that runs at merge time, not on every push. ### What tools are used to build CI/CD pipelines? The CI runner is one of GitHub Actions, GitLab CI, CircleCI, Buildkite, Jenkins, or (in monorepos) a build-tool-driven setup around Bazel. The deploy step usually talks to a container registry (ECR, GCR, Docker Hub) and an orchestrator (Kubernetes, ECS, Cloud Run) or a platform-as-a-service (Vercel, Render, Fly.io). ### What is the difference between a CI/CD pipeline and a build pipeline? A build pipeline produces an artifact: compile the code, package it, push the image. A CI/CD pipeline does that and also runs the tests, gates the merge on success, and pushes the artifact to staging and production. The build pipeline is one stage of a CI/CD pipeline. ## A pipeline is only useful if it is fast, trustworthy, and never quietly broken. CI Insights tells you which stages are slowing the pipeline down, which tests are unreliable, and where the build is failing. The merge queue tests every PR against the actual future state of main before it merges, so two green PRs cannot break it together. [See CI Insights](https://mergify.com/product/ci-insights)[See the merge queue](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/learn/circleci/test-results CircleCI # Test results in CircleCI CircleCI shows your tests once your runner writes JUnit XML and a `store_test_results` step uploads it. Here is a config that works, what the Tests tab and Test Insights do with those files, how CircleCI flags and reruns flaky tests, and how to send the same results to Mergify. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated October 30, 2026 ## In one paragraph Make your test runner write JUnit XML into a directory, and add `store_test_results` with that directory as its `path`. The step runs even when the tests failed. CircleCI then lists each job's tests in the Tests tab, uses their durations to split tests across parallel containers when your test command splits by timing, and builds Test Insights over time, where a test that failed and passed on the same commit within 14 days is labeled FLAKY. Rerunning only the failed tests goes through Smarter Testing's `circleci testsuite run`. ## store\_test\_results in .circleci/config.yml This config runs a pytest suite and stores its results: ``` version: 2.1 jobs: test: docker: - image: cimg/python:3.13 steps: - checkout - run: pip install -r requirements.txt - run: name: Run tests command: | mkdir -p test-results pytest --junitxml=test-results/junit.xml - store_test_results: path: test-results workflows: main: jobs: - test ``` `path` is relative to the job's working directory and can be a directory (CircleCI reads every JUnit XML file in it and in its subdirectories) or a single file. Three details trip people up: the directory must not be hidden, so `.test-results` does not work; CircleCI does not expand variables in it, so `$CIRCLE_WORKING_DIRECTORY/test-results` is read literally; and nothing is stored if the runner writes no XML, which most runners do not do by default. pytest takes `--junitxml`, Vitest and Playwright have a built-in `junit` reporter, Jest needs `jest-junit`, RSpec needs `rspec_junit_formatter`, and Go needs `gotestsum`. The [JUnit XML guide](https://mergify.com/learn/junit-xml) has the exact command for each. You do not need `when: always` on `store_test_results`. A failed `run` step skips the `run` steps after it, but `store_test_results` and `store_artifacts` always run, so a failed build still has its results. They are skipped only when the job is canceled or times out. ## What CircleCI does with your results
Tests tabThe passing and failing tests of each job, on the job page.
Test splitting by timingWith parallelism and a test command that splits by timing, such as circleci tests run --split-by=timings, CircleCI uses the durations in your stored results to spread tests across containers. parallelism on its own runs the whole suite in every container.
Test InsightsAcross runs of the project: the most failed tests, the slowest ones, and tests labeled FLAKY. Organizations whose slug starts with circleci/ (rather than github/ or bitbucket/) do not get it.
Insights APIThe same test data over CircleCI's API v2, for your own dashboards.
From CircleCI's [test data](https://circleci.com/docs/guides/test/collect-test-data/), [parallelism](https://circleci.com/docs/guides/optimize/parallelism-faster-jobs/) and [Test Insights](https://circleci.com/docs/guides/insights/insights-tests/) documentation, as of November 2026. ## Flaky tests in CircleCI: the label and the rerun Test Insights labels a test **FLAKY** when it failed and passed on the same commit within a 14-day window. In practice that is usually a test someone reran: the workflow failed, an engineer clicked rerun, the test passed. The label is useful for finding those tests. It does not change what they do to your build: a FLAKY test keeps failing the jobs it fails in until someone fixes it. For the rerun itself, CircleCI's current answer is Smarter Testing. You describe how to discover and run your tests in `.circleci/test-suites.yml`, replace your test command with `circleci testsuite run`, and from then on _Rerun workflow from failed_ runs only the tests that failed instead of the whole suite, at no extra charge. The older _Rerun failed tests_ button, built on `circleci tests run`, is deprecated and only shows on projects that still use that command. Smarter Testing can also retry failed tests within the same run: set `max-auto-rerun` in `test-suites.yml`, and if every failed test passes on a retry, the job succeeds. That is charged in credits above your plan's allowance. It is also where flaky tests start to disappear from view: a job that went green on its second attempt looks like any other green job, so keep an eye on what got retried. The [rerun only failed tests](https://circleci.com/docs/guides/test/rerun-only-failed-tests/) and [auto rerun](https://circleci.com/docs/guides/test/auto-rerun-failed-tests/) pages have the setup. ## Send the same results to Mergify Test Engine Test Engine keeps every test's result from every run, classifies each test as healthy, flaky or broken from that history, and can quarantine the flaky ones so they keep running without failing your build. It reads the same results CircleCI does, and your `store_test_results` step stays. Start with a Mergify application key with the `ci` scope, stored as `MERGIFY_TOKEN` in a project environment variable (every job gets it) or in a context (add `context` to each test job in `workflows`). Never put it in the config file. **With a test plugin** (pytest-mergify, @mergifyio/vitest, @mergifyio/playwright, @mergifyio/jest, rspec-mergify), install it and turn it on in your test runner, as its [setup page](https://docs.mergify.com/test-engine/test-frameworks) shows: pytest-mergify works once installed, Vitest needs `MergifyReporter` in its `reporters`, Playwright and Jest wrap their configuration in `withMergify`, and RSpec needs `--require rspec_mergify`. The CircleCI config above does not change: CircleCI sets `CI` and the `CIRCLE_*` variables the plugin reads, and each job reports under its own name. **With any other framework**, upload the JUnit XML it already writes with the Mergify CLI, in the same `run` step as the tests: ``` version: 2.1 jobs: test: docker: - image: cimg/go:1.25 steps: - checkout - run: name: Install gotestsum and the Mergify CLI command: | go install gotest.tools/gotestsum@v1.13.0 curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh - run: name: Run tests and upload them to Mergify command: | mkdir -p test-results exit_code=0 gotestsum --junitfile test-results/junit.xml ./... || exit_code=$? if [ -n "${MERGIFY_TOKEN:-}" ]; then mergify ci junit-process --test-exit-code "$exit_code" test-results/junit.xml else exit "$exit_code" fi - store_test_results: path: test-results workflows: main: jobs: - test ``` CircleCI runs each step with `bash -eo pipefail`, so the test command on its own line would end the step before the upload. `|| exit_code=$?` records the exit code instead, `--test-exit-code` passes it on so a runner that crashed is not read as a clean report, and `junit-process` exits last, so its status becomes the step's. The `if` covers builds without a token: there the step exits with the tests' own code, so failing tests still fail the job. Builds of pull requests from forks are those builds: they get no project variables or contexts unless you enable _Pass secrets to builds from forked pull requests_. The plugins then upload nothing and your tests run as usual, and `junit-process`, which would fail the step without a token, is skipped. CircleCI publishes no base branch for a pull request build either, so the plugins and `junit-process` ask Mergify which open pull request has the build's commit as its head, and read the build as one into that pull request's base. There is nothing to configure, and [Prevention](https://docs.mergify.com/test-engine/prevention), which reruns the tests a pull request adds to catch the flaky ones, works on those builds. A build Mergify knows no pull request for, such as one pushed before the pull request was opened, is read like a branch build, and so is a build of your default branch. This lookup ships with version 2026.11.1 of the plugins and the CLI. The [CircleCI setup docs](https://docs.mergify.com/ci-insights/setup/circleci) have the details. ## FAQ ### What does store\_test\_results do in CircleCI? It uploads the JUnit XML files your tests wrote, from a directory or a single file, so CircleCI can parse them. That feeds the Tests tab on the job page, timing-based test splitting, Test Insights and its flaky test detection. CircleCI does not collect test results on its own: without the step and a JUnit reporter, it only knows whether the job passed. ### Does store\_test\_results run when the tests fail? Yes. store\_test\_results and store\_artifacts always run, even after a failed step, so you do not need a when: always on them. They do not run when the job is canceled or hits its timeout; CircleCI's teardown steps cover that case. ### Why does the Tests tab say no test results were found? The path points at nothing CircleCI can parse. Check that the path is relative to the working directory, is not a hidden directory (.reports does not work), and that your runner actually wrote JUnit XML there: pytest needs --junitxml, Jest needs jest-junit, RSpec needs rspec\_junit\_formatter. CircleCI does not expand variables in the path either. ### How does CircleCI detect flaky tests? Test Insights labels a test FLAKY when it failed and passed on the same commit within a 14-day window. That usually means someone reran a workflow and the test passed the second time. The label is informational: a FLAKY test still fails the jobs it fails in. ### How do I rerun only the failed tests in CircleCI? Run your tests through Smarter Testing's circleci testsuite run, configured in .circleci/test-suites.yml, then use Rerun workflow from failed: only the tests that failed run again, at no extra charge. The older Rerun failed tests button, built on circleci tests run, is deprecated. Auto rerun retries failed tests within the same run, and is charged in credits above your plan's allowance. Last updated: October 30, 2026 ## Your CircleCI tests, with a failure rate Keep store\_test\_results. Add per-test history, flaky detection and quarantine to the same config. Free for up to 5 users. [Get started](https://dashboard.mergify.com/)[See how Test Engine works](https://mergify.com/product/test-engine) --- Source: https://mergify.com/learn/continuous-integration Guide # Continuous Integration Continuous integration is the practice of merging every engineer's work into one shared branch many times a day, with an automated build and test on every change. The mechanics are well known. What separates teams that get value from CI from teams that just pay for it is what they do with the signal. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated May 19, 2026 ## In one paragraph Every change an engineer makes gets pushed, built, and tested automatically, many times a day. A green pipeline means the change is safe to integrate. A red pipeline blocks the merge. The point is to surface integration problems immediately, when one engineer can fix them in an hour, instead of at the end of a release cycle when the whole team has to. CI is the foundation that [trunk-based development](https://mergify.com/learn/trunk-based-development), continuous delivery, and modern merge queues are built on. ## The problem CI solves flowchart LR Dev1\["Engineer A
2 weeks of work"\] Dev2\["Engineer B
2 weeks of work"\] Dev3\["Engineer C
2 weeks of work"\] IntegrationDay\["Integration
day"\] Broken\["main
BROKEN
for days"\] Dev1 --> IntegrationDay Dev2 --> IntegrationDay Dev3 --> IntegrationDay IntegrationDay --> Broken style Broken fill:#FDECEA,stroke:#E53935,color:#1A1D24 style IntegrationDay fill:#FFF4E5,stroke:#F27B2A,color:#1A1D24 Pre-CI integration: three engineers work in isolation for two weeks, then try to combine the work in one go. The conflicts are not in the code, they are in the assumptions each engineer made about a codebase that has since changed underneath them. Before CI, teams shipped on a calendar. Engineers worked on long-lived branches for weeks at a time. At the end of the cycle, somebody (the unlucky release engineer) merged all the branches back into one main branch and spent the next several days debugging the result. The phrase "integration hell" is older than most engineers using it now. It described a real week on a real calendar. The cost of integration is not linear in branch lifetime, it is exponential. A two-day branch is a regular merge. A two-week branch is a project. A two-month branch is the kind of thing teams quietly abandon. The reason is that integration is about reconciling two diverging worlds: every day the branch stays open, both worlds change, and the surface area of the eventual merge grows. Continuous integration cuts the problem at its source. Branches do not survive long enough for the world around them to change. The integration cost stays small because integration happens constantly, not on a schedule. ## How CI works in practice flowchart LR Push\["Push to branch"\] Build\["Build"\] Test\["Run tests"\] Signal{"Pass?"} Green\["Mergeable ✓"\] Red\["Blocked
Fix the branch"\] Push --> Build --> Test --> Signal Signal -->|yes| Green Signal -->|no| Red style Green fill:#E6F8F2,stroke:#1CB893,color:#1A1D24 style Red fill:#FDECEA,stroke:#E53935,color:#1A1D24 The CI loop: every change is built and tested before it can be considered mergeable. The mechanics are unremarkable. An engineer pushes a commit to a branch. A CI tool picks up the push, runs a pipeline, and reports the result back as a status check on the pull request. The pipeline almost always contains the same steps in roughly the same order: clone the repository, install dependencies, build the code, run linters, run tests, report. The pipeline is defined as code, usually in a YAML file checked into the repo (`.github/workflows/ci.yml`, `.gitlab-ci.yml`, `.circleci/config.yml`, and so on). That choice matters: the pipeline lives in the same git history as the code, which means changing the build is a regular pull request like any other. ### A trustworthy signal For CI to be useful, a green pipeline has to mean something. If half the failures on main are flaky tests that pass on retry, engineers learn to ignore the signal. Once they ignore it, the signal is gone, and CI becomes a tax that everyone pays without anyone benefiting. Flaky-test tooling like [Test Engine](https://mergify.com/product/test-engine) exists because this failure mode is universal. ### Fast enough to use If the pipeline takes 45 minutes, nobody is going to push four times a day. The pipeline has to fit inside the cycle of "make a change, push, wait, react." Ten to fifteen minutes is the upper bound where most teams stay productive. Beyond that, the levers are well known: parallelize the test suite, run only the tests affected by the change (selective testing), and [split heavy checks into a second stage that only runs at merge time](https://mergify.com/learn/merge-queue/two-step-ci). ### Required on the way to main A passing pipeline is only useful if it is enforced. Most teams set CI as a required check, so a pull request cannot merge until the pipeline is green. Without that enforcement, "CI is broken on main" becomes a regular condition rather than an incident, and the whole loop unravels. ## CI, continuous delivery, continuous deployment Three terms, often used interchangeably, that describe three different things. | Practice | What it covers | Who runs it | | --- | --- | --- | | Continuous integration | Every change is built and tested in main automatically. | Every engineering team that uses a CI tool. | | Continuous delivery | Main is always in a state that could be deployed to production. Deployment itself is a manual click. | Most production SaaS. | | Continuous deployment | Every green commit is deployed to production automatically. | A smaller set: companies with strong test coverage, feature flags, and on-call discipline. | Continuous integration is the prerequisite. Without an always-green main, there is nothing safe to deliver and certainly nothing safe to deploy. The other two practices are about what the team decides to do once that prerequisite is in place. ## Where CI quietly stops working Most teams that have CI do not have a CI problem in the obvious sense. The pipelines run, the YAML is checked in, the green ticks show up on pull requests. The problem is that the signal has gradually decayed and nobody noticed. ### Flaky tests Tests that fail on one run and pass on the next destroy trust faster than anything else. After a week of retries, engineers learn that red is not necessarily red, and they start clicking "rerun" by reflex. Once that habit forms, every real failure goes through "is this real, or is this flaky?" first, and CI is no longer a signal, it is a question. The fix is to detect flakes systematically, quarantine them out of the required check, and assign the cleanup. See the [flaky tests guide](https://mergify.com/learn/flaky-tests) for the framework-specific patterns and [Test Engine](https://mergify.com/product/test-engine) for what it looks like to automate this. ### A slow pipeline When CI takes 30 minutes, pushing a change costs 30 minutes of attention. Engineers context-switch, lose the thread, come back to a failure and have to remember what they were doing. The healthy ceiling is around 10 to 15 minutes. Past that, parallelization, selective testing, and [two-step CI](https://mergify.com/learn/merge-queue/two-step-ci) are the levers. ### Two green PRs that break main together This one surprises teams the most. Two pull requests pass CI in isolation, both green, both reviewed. They land on main back to back. Main is red the moment the second one merges, because the two changes were tested against an older version of main, not against each other. A function got renamed in PR #1 while a caller was added in PR #2. Neither pipeline saw both changes at once. The fix is a [merge queue](https://mergify.com/learn/merge-queue) that tests each PR against the actual future state of main before it merges. CI alone cannot catch this, by design. ### No visibility on what is failing CI emits a green or red tick. That is not enough to run a real engineering org. Teams that take CI seriously look at retry rate, time to green, flake rate per test, slowest tests, and where in the pipeline failures concentrate. Without that data, "CI is slow" and "CI is flaky" stay as vague complaints in retros and nothing gets fixed. [CI Insights](https://mergify.com/product/ci-insights) exist for this, but even a basic dashboard built on top of the CI tool's API is better than guessing. ## The CI tools landscape For most teams the choice of CI tool is settled by the choice of forge: - **GitHub Actions**. The default for new projects on GitHub. Tight integration with pull requests, large marketplace of actions, scales to most workloads. The default choice unless something pulls you elsewhere. - **GitLab CI**. The native pipeline runner for GitLab. Mature, opinionated, well integrated with merge requests and GitLab's wider DevOps surface. - **CircleCI**. Independent CI provider. Strong macOS support, good caching, used heavily by mobile and cross-platform teams. - **Buildkite**. Self-hosted runners with a hosted control plane. The pattern of choice for teams that want CI on their own hardware (GPU, large machines, on-prem secrets) without running the orchestration themselves. - **Jenkins**. The oldest of the modern CI tools. Still pervasive in enterprise. Powerful and extensible. Operationally heavy in a way that newer tools are not. - **Bazel and other build-tool-driven setups**. Inside large monorepos (Google, Meta, Uber, Stripe), CI is often a thin shell around a build tool that does selective testing, remote caching, and dependency tracking. The CI tool runs Bazel; Bazel decides what to run. Once the CI tool is picked, the interesting questions are not about the tool. They are about the pipeline (what does it run, how fast, how reliably) and about the surrounding system (merge queue, flake detection, observability) that turns the pipeline output into something useful. ## If you are setting CI up for the first time A reasonable order to get to a useful baseline without overbuilding. 1. **Pick the CI tool that matches your forge.** GitHub Actions for GitHub, GitLab CI for GitLab. Switching later is annoying but possible, so do not over-evaluate. 2. **Start with build plus linters plus unit tests.** A green pipeline should mean "the code compiles, the formatters and linters are happy, and the fast tests pass." That is enough to start trusting the signal. 3. **Make it a required check.** Block merges to main on a green pipeline. If you skip this step, the signal will atrophy within a few weeks. 4. **Add integration and end-to-end tests next, in a slower stage.** Many teams discover too late that running Playwright on every push costs more in time and money than they can afford. Separate fast checks (run on every PR push) from heavy checks (run once at merge time). That is the basic shape of [two-step CI](https://mergify.com/learn/merge-queue/two-step-ci). 5. **Add a merge queue as soon as you have two engineers merging in parallel.** The cost of waiting until main breaks is higher than the cost of setting it up early. 6. **Instrument the pipeline.** Failure rates, retry rates, time to green, flake rate per test. You need numbers to know which step to speed up next. ## FAQ ### What is continuous integration? Continuous integration is the practice of merging every engineer's work into one shared branch many times a day, with an automated build and test running on every change. The point is to find integration problems immediately, when they are still cheap to fix, instead of accumulating them until release week. ### What is the difference between continuous integration and continuous delivery? Continuous integration is about merging code: every change goes into main, every change is built and tested. Continuous delivery is about shipping it: main is always in a state that could be deployed to production at any time. Continuous deployment goes one step further and actually pushes every green commit to production automatically. CI is the foundation the other two are built on. ### What is a CI pipeline? A CI pipeline is the sequence of automated steps that runs on every change: typically clone, install dependencies, build, run linters and tests, report results back to the pull request. Most pipelines are defined as code in a file like .github/workflows/ci.yml, .gitlab-ci.yml, or .circleci/config.yml. ### What is a CI tool? A CI tool is the system that executes the pipeline. The common ones today are GitHub Actions, GitLab CI, CircleCI, Buildkite, Jenkins, and (still) Bazel-driven custom setups inside large monorepos. The tool runs the steps. The discipline of integrating constantly is what makes it continuous. ### Why is continuous integration important? Because integration cost compounds. A two-day branch is roughly a regular merge. A two-week branch is a project. Without CI, integration is the work nobody does until it has to be done all at once, usually at the worst possible time. CI moves that cost from a periodic crisis to a constant background hum. ### Does CI require automated tests? In a meaningful sense, yes. The whole point of CI is a fast, trustworthy signal on every change. A CI pipeline that only builds does not give you that. The minimum is a build plus enough automated tests that a green pipeline actually means something. Adding tests over time is a normal part of CI adoption. ### Does continuous integration replace code review? No. CI verifies that the code builds and the tests pass. Code review verifies that a human thinks the change is correct, well structured, and consistent with the rest of the codebase. They check different things and most engineering orgs require both before merge. ### How does CI work with pull requests? Every PR triggers the pipeline against the PR's commits. The result is reported back as a check on the PR (green tick or red cross). Most teams set the green CI as a required check, so a PR cannot merge until the pipeline passes. Once it merges, CI runs again on main to confirm the merged result is still green. ## CI gives you a signal. Mergify is what you do with it. CI Insights surfaces flakes, slow tests, and retry storms. The merge queue is how every green PR stays green when it lands on main. The two together are what makes continuous integration actually useful. [See CI Insights](https://mergify.com/product/ci-insights)[See the merge queue](https://mergify.com/product/merge-queue) --- Source: https://mergify.com/learn/flaky-tests # Flaky tests. Named, fixed, quarantined. Flaky tests are not random noise. They follow patterns, and the patterns are specific to each test framework. Once you name the pattern, you can kill it. This guide covers every major framework, with the fixes we have seen work across thousands of CI pipelines. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated May 19, 2026 [Try Mergify Test Engine](https://dashboard.mergify.com/)[See the product](https://mergify.com/product/test-engine) ## Pick your framework Every framework has its own flakiness patterns. Start with yours. [ Jest Fake timers, snapshot drift, and the async traps behind flaky Jest suites. Read the guide →](https://mergify.com/learn/flaky-tests/jest)[ pytest Fixture teardown races, xdist ordering, and Hypothesis seed non-determinism. Read the guide →](https://mergify.com/learn/flaky-tests/pytest)[ Cypress cy.wait() races, network stubbing, and retry-ability gotchas. Read the guide →](https://mergify.com/learn/flaky-tests/cypress)[ Playwright Auto-waiting versus explicit waits, trace-level debugging, and retry config. Read the guide →](https://mergify.com/learn/flaky-tests/playwright)[ Vitest Thread vs fork pool, mock hoisting, and shared-state snapshot flakiness. Read the guide →](https://mergify.com/learn/flaky-tests/vitest)[ RSpec Order-dependent specs, database\_cleaner strategies, and rspec-retry pitfalls. Read the guide →](https://mergify.com/learn/flaky-tests/rspec)[ Minitest Parallelization races, fixture loading, and test isolation in Rails suites. Read the guide →](https://mergify.com/learn/flaky-tests/minitest)[ Go t.Parallel() races, goroutine leaks, and nondeterministic map iteration. Read the guide →](https://mergify.com/learn/flaky-tests/go)[ Rust Test-thread races, async runtime drift, and cargo test parallelism. Read the guide →](https://mergify.com/learn/flaky-tests/rust)[ JUnit Method ordering, Surefire forking, and shared static state in Spring tests. Read the guide →](https://mergify.com/learn/flaky-tests/junit)[ TestNG Dependency groups, parallel methods, and retry analyzers done wrong. Read the guide →](https://mergify.com/learn/flaky-tests/testng)[ PHPUnit Test isolation, database transactions, and dataProvider non-determinism. Read the guide →](https://mergify.com/learn/flaky-tests/phpunit)[ Pest Higher-order tests, parallel execution, and the Laravel testing stack. Read the guide →](https://mergify.com/learn/flaky-tests/pest)[ NUnit ParallelizableAttribute, TestContext races, and fixture reuse pitfalls. Read the guide →](https://mergify.com/learn/flaky-tests/nunit)[ MSTest Assembly initialization, DeploymentItem races, and parallel test settings. Read the guide →](https://mergify.com/learn/flaky-tests/mstest) Setting it up on your CI: [flaky tests on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions). ## Kill flaky tests across every framework in your stack. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/cypress [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/Cypress # Flaky tests in Cypress. Named, fixed, and quarantined. Flaky Cypress suites are not random. They follow patterns: cy.wait races, intercept-too-late, detached DOM, broken retry-ability, session leakage. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Published April 2026 Example PR comment from the Mergify bot detecting a flaky Cypress test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky Cypress suite: cy.wait(ms) racing the page, cy.intercept registered after the request fires, detached DOM errors after re-render, synchronous .then() breaking command-chain retry-ability, session leakage without cy.session(), CSS animations stealing clicks, server state contamination from previous specs, and Cypress retries config hiding real bugs. Each one has a minimal reproducer, a fix in stock Cypress, and a Mergify Test Engine detection strategy below. ## Why are Cypress tests flaky? Cypress's command queue is the source of its expressiveness and the source of its flakes. Every `cy.something()` call is a deferred command, not a synchronous action: Cypress builds a chain, then runs it, with [automatic retry on the last command](https://docs.cypress.io/guides/core-concepts/retry-ability) until an assertion passes or the timeout hits. The model is brilliant when you stay inside the chain. The moment you drop into a native `.then()` with synchronous logic, retry-ability vanishes and your test becomes timing-sensitive. Layer on the browser context: cookies, local storage, service workers, and an in-page network layer that [`cy.intercept`](https://docs.cypress.io/guides/guides/network-requests) only sees if the route is registered before the request fires. CSS animations make the click target move under your cursor. [cy.origin](https://docs.cypress.io/api/commands/origin) guards keep tests from accidentally crossing domains, but only if you remember to declare them. Each rule has a clean fix once you can name the failure mode. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of Cypress suites: cy.wait(ms) racing the page, cy.intercept registered after the request fires, detached DOM errors after re-render, synchronous .then() breaking command-chain retry-ability, session leakage without cy.session(), CSS animations stealing clicks, server state contamination from previous specs, and Cypress retries config hiding real bugs. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01cy.wait(ms) racing the page](#pattern-cy-wait-ms-racing-the-page) 2. [02cy.intercept registered after the request fires](#pattern-cy-intercept-registered-after-the-request-fires) 3. [03Detached DOM errors after re-render](#pattern-detached-dom-errors-after-re-render) 4. [04Synchronous .then() breaking retry-ability](#pattern-synchronous-then-breaking-retry-ability) 5. [05Session leakage without cy.session()](#pattern-session-leakage-without-cy-session) 6. [06CSS animations stealing clicks](#pattern-css-animations-stealing-clicks) 7. [07Server state contamination from previous specs](#pattern-server-state-contamination-from-previous-specs) 8. [08Cypress retries config hiding real bugs](#pattern-cypress-retries-config-hiding-real-bugs) Pattern 1 ### cy.wait(ms) racing the page Symptom. A test passes locally, fails on the slower CI runner, with \`cy.click()\` failing because the target was not in the DOM yet. Root cause. [`cy.wait(ms)`](https://docs.cypress.io/api/commands/wait#Wait-for-an-arbitrary-period-of-milliseconds) is a hardcoded sleep. It does nothing useful: Cypress already retries assertions and queries until the page settles or the timeout hits. A magic number works when the local machine renders in 200ms and breaks when CI takes 1100ms. Worse, it adds dead time to every passing run, slowing the suite down for the same reason it never actually waited for the right thing. ``` it("opens the modal", () => { cy.visit("/dashboard"); cy.get("button.open-modal").click(); cy.wait(500); // hope the modal is open by now cy.get("input[name=email]").type("user@example.com"); // fails if it isn't }); ``` Fix. Wait for what you actually need: an alias from `cy.intercept`, an assertion that an element exists, or a state change. Cypress retries the assertion on its own, with the configured timeout, so the test is fast when fast and patient when slow. ``` it("opens the modal", () => { cy.visit("/dashboard"); cy.get("button.open-modal").click(); // Cypress polls until the modal is in the DOM, with default timeout cy.get("[role=dialog]").should("be.visible"); cy.get("input[name=email]").type("user@example.com"); }); ``` With Mergify. Test Engine notices that the same spec only fails on the slower CI runner pool and never on the laptop pool. The dashboard tags the resource sensitivity so the timing assumption is the obvious place to look. Pattern 2 ### cy.intercept registered after the request fires Symptom. A test that asserts on a stubbed network response fails with \`cy.wait('@alias') timed out\`, but only intermittently and never on rerun. Root cause. [`cy.intercept`](https://docs.cypress.io/api/commands/intercept) only catches requests that fire after the interceptor is registered. Calling `cy.visit()` first and `cy.intercept` after means a fast browser dispatches the XHR before Cypress installs the route. The request goes through to the real server, the alias never resolves, and the test waits until the timeout. ``` it("shows the user", () => { cy.visit("/profile"); // fires GET /api/me immediately cy.intercept("GET", "/api/me", { id: "u-1", name: "Rémy" }).as("me"); cy.wait("@me"); // times out: the request already happened }); ``` Fix. Always register the interceptor before the action that triggers the request. The standard pattern is `intercept` first, then `visit` or whatever event causes the fetch. ``` it("shows the user", () => { cy.intercept("GET", "/api/me", { id: "u-1", name: "Rémy" }).as("me"); cy.visit("/profile"); cy.wait("@me"); // resolves cleanly }); ``` With Mergify. Test Engine records the failure URL and surfaces specs whose only failure mode is \`cy.wait timeout\`. The dashboard groups them so the registration-order pattern is easy to spot across the suite. Pattern 3 ### Detached DOM errors after re-render Symptom. A spec fails with \`CypressError: Element is detached from the DOM\` on a click that worked the previous five runs. Root cause. Cypress queries the DOM, finds an element, and queues an action against it. If the page re-renders between the query and the action (a React state change, an SSR hydration finishing, an `v-if` flipping), the queried node is still in memory but no longer attached to the document. Cypress refuses to act on a detached node. ``` it("submits the form", () => { cy.get("button.submit") .then((el) => { // synchronous Cypress code path; queries once, holds the reference el.trigger("click"); // by the time this fires, the form has re-rendered with a fresh button // → "Element is detached from the DOM" }); }); ``` Fix. Stay inside the Cypress command chain so each command requeries on retry. Avoid grabbing a jQuery handle and acting on it across an async gap. When the chain itself crosses a re-render, scope the query to a stable parent so the requery picks the new child. ``` it("submits the form", () => { cy.get("form#checkout") .find("button.submit") .click(); // requeries 'button.submit' inside the form on retry }); ``` With Mergify. Test Engine flags specs that fail only with detached-DOM errors as re-render races, distinct from logic failures. The dashboard groups them by the component name in the query selector so the offending re-render is easy to find. Pattern 4 ### Synchronous .then() breaking retry-ability Symptom. A spec works against a mock and fails against a real backend, with the assertion inside a \`.then()\` reading stale data. Root cause. [Cypress only retries the last command in a chain](https://docs.cypress.io/guides/core-concepts/retry-ability). Wrapping a query in `.then()` and asserting inside the callback freezes the value at the moment of the callback's first execution. If the data was not ready yet, the assertion fails and Cypress never retries it because `.then()` succeeded. ``` it("loads the user count", () => { cy.get("[data-test=count]") .then(($el) => { // $el captured once. If the count was '0' on first paint, that's what // we assert against, even though the real value lands 100ms later. expect($el.text()).to.eq("42"); }); }); ``` Fix. Move the assertion into a chained `.should()` so Cypress retries the whole query+assertion until it passes or times out. Reach for `.then()` only when you need to capture a value for use outside Cypress. ``` it("loads the user count", () => { cy.get("[data-test=count]").should("have.text", "42"); }); ``` With Mergify. Test Engine detects the mock-vs-real-backend signature: the same spec is reliable when the network is stubbed and unreliable when it hits a slower service. The dashboard surfaces the dependency so the retry-ability mistake is easy to locate. Pattern 5 ### Session leakage without cy.session() Symptom. A test fails on cold runs and passes on rerun, or a logout spec leaves the suite in a state the next spec did not expect because both share a hand-rolled login command. Root cause. Cypress clears cookies, local storage, and session storage for the active origin between tests by default. A hand-rolled login command that calls `cy.request` and writes a token via `cy.window().its("localStorage")` reruns the full login on every test, which is slow on suites and brittle when login itself is racy. [`cy.session()`](https://docs.cypress.io/api/commands/session) exists to cache validated login state per key across tests; skipping it forces every test to redo the work and amplifies any flake in the auth path. ``` Cypress.Commands.add("login", (user) => { cy.request("POST", "/api/login", user).then((res) => { cy.visit("/"); cy.window().then((win) => { win.localStorage.setItem("token", res.body.token); }); }); }); beforeEach(() => { cy.login({ email: "user@example.com", password: "..." }); // Login fires for every test. When the auth service is intermittently slow, // any spec can fail; rerun usually wins the race and the bug stays hidden. }); ``` Fix. Wrap the login work in `cy.session(key, setup, options)`. Cypress runs the setup once per key, snapshots cookies + storage, and rehydrates them on every later test that requests the same key. Pass a `validate` callback so a stale cached session is rebuilt instead of silently reused. ``` Cypress.Commands.add("login", (user) => { cy.session( user.email, () => { cy.request("POST", "/api/login", user).then((res) => { cy.visit("/"); cy.window().then((win) => { win.localStorage.setItem("token", res.body.token); }); }); }, { validate: () => { cy.request("/api/me").its("status").should("eq", 200); }, }, ); }); ``` With Mergify. Test Engine catches the order-dependent signature: a spec only fails after a specific other spec has run. The dashboard groups failures by the predecessor spec so the missing cy.session is easy to identify. Pattern 6 ### CSS animations stealing clicks Symptom. A \`cy.click()\` on a button fails with \`actionability\` errors mentioning \`animation\` or \`position\` changing between actionability checks. Root cause. [Cypress measures the element's position before clicking](https://docs.cypress.io/guides/core-concepts/interacting-with-elements#Animations). If the element is mid-animation (a slide-in, a fade, a CSS transition triggered by the previous click), Cypress sees the position change between samples and refuses to act, fearing it would click the wrong target. Locally the animation finishes faster than Cypress measures; on CI the throttled CPU stretches the animation past the actionability threshold. ``` it("dismisses the toast", () => { cy.get("button.show-toast").click(); // toast slides in over 300ms cy.get(".toast button.dismiss").click(); // CypressError: '.toast button.dismiss' is animating; aborting click }); ``` Fix. Disable animations in the test environment via a global CSS override or a config flag. For a one-off, force the click with the `{ force: true }` option, but only after you have checked the animation is the actual cause and not a real layout bug. ``` // cypress/support/e2e.ts beforeEach(() => { cy.get("head").invoke( "append", Cypress.$(''), ); }); ``` With Mergify. Test Engine tags actionability-related failures distinctly from assertion failures. The dashboard groups specs whose failures all mention 'animating' so the global config fix is one commit instead of 30. Pattern 7 ### Server state contamination from previous specs Symptom. A spec asserts on a clean dashboard and fails because rows from the previous spec's seed data are still in the test database. Root cause. Cypress is a browser tool: it does not own the backend. A spec that `POST /users` leaves the user in the database for whichever spec runs next. If the suite assumes a clean state at the top of every spec without resetting the backend, the order in which specs run silently changes which assertions pass. ``` // spec-a.cy.ts it("creates a user", () => { cy.request("POST", "/api/users", { name: "Rémy" }); cy.visit("/users"); cy.get("table tr").should("have.length", 1); }); // spec-b.cy.ts (runs after spec-a in alphabetical order) it("starts with no users", () => { cy.visit("/users"); cy.get("table tr").should("have.length", 0); // fails: 1 row from spec-a }); ``` Fix. Reset the backend before each spec via a test-only endpoint or a database task. [`cy.task()`](https://docs.cypress.io/api/commands/task) runs Node code in the Cypress process, which is the right place for "truncate the test database" hooks. ``` // cypress.config.ts export default defineConfig({ e2e: { setupNodeEvents(on) { on("task", { resetDb: () => truncateAllTables() }); }, }, }); // cypress/support/e2e.ts beforeEach(() => cy.task("resetDb")); ``` With Mergify. Test Engine groups failures whose only signature is row-count or unique-constraint violations. The dashboard surfaces them as state-leakage candidates so the missing reset hook is the obvious fix. Pattern 8 ### Cypress retries config hiding real bugs Symptom. Your suite is green. A user reports a bug your specs were supposed to catch. Root cause. [`retries: { runMode: 3 }`](https://docs.cypress.io/guides/guides/test-retries) in `cypress.config.ts` reruns failing specs up to three times in CI and reports the last result. A real race that loses on attempt 1 and wins on attempt 2 gets reported as green. The bug is still there. The pipeline has decided not to look at it. ``` // cypress.config.ts (please don't) export default defineConfig({ retries: { runMode: 3, openMode: 0 }, }); ``` Fix. Do not retry at the framework level. When a spec is genuinely flaky, fix it. When the fix takes longer than a session, quarantine it instead. That keeps the signal visible without blocking the merge queue. With Mergify. Test Engine reruns at the CI level with attempt-level result tracking. You see that a spec passed on attempt 2 of 3, which is exactly the information Cypress retries throws away. Quarantine kicks in once the pattern is clear. Detection ## How to detect flaky Cypress tests in CI Add the cypress-junit-reporter package, point Cypress at it on every CI run, and upload the resulting XML to Mergify with a one-line CLI call. Test Engine builds a confidence score for every spec on your default branch. PR runs are compared against that baseline. Anything inconsistent gets flagged in a PR comment before the author merges. mergify ci ``` # 1. Add the JUnit reporter npm install --save-dev cypress-junit-reporter # 2. Emit JUnit XML on every CI run cypress run --reporter cypress-junit-reporter \ --reporter-options "mochaFile=junit.xml,toConsole=false" # 3. Upload the result (once, in CI) curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh mergify ci junit-process junit.xml ``` Prevention ## How to prevent flaky Cypress tests from merging On every PR, Mergify reruns the tests whose confidence is below threshold, without Cypress retries config touching your config. The PR gets a comment naming the unreliable tests, their confidence history, and whether the failure on this PR is new or historical noise. Authors fix the real bugs before merge instead of re-running CI until it passes. Quarantine ## How to quarantine flaky Cypress tests without skipping them Once a Cypress test is confirmed flaky, Test Engine quarantines it. The test still runs in the suite, no \`it.skip()\` rewrite required, but its result no longer blocks merges or marks the pipeline red. When the pass rate on main recovers, quarantine lifts automatically and the test goes back to being load-bearing. Your CI ## Run it on your CI The Cypress setup for Test Engine on each CI, how its retries compare with the CI's own reruns, and what makes Cypress flake on that CI's runners. - [Cypress on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions/cypress) ## Want to see which Cypress specs in your repo are already flaky? Works with cypress-junit-reporter or any JUnit-compatible Cypress reporter. Setup takes under five minutes. [Book a discovery call](https://mergify.com/discovery) ## Frequently asked questions Why are my Cypress tests flaky in CI but pass locally? Your laptop and your CI runner differ in CPU, network latency, and animation throttling. Specs that depend on \`cy.wait(ms)\` magic numbers, register \`cy.intercept\` after the request fires, or click through CSS animations lose the race under CI's tighter resource budget. Reproduce locally with \`cypress run --browser chrome --headless\` (CI's actual mode), or throttle your CPU in DevTools to mirror the CI runner, then fix the underlying timing assumption before pushing. How do I detect flaky Cypress tests? Cypress alone cannot tell flaky from broken since each run gives one data point per test. You need to run the same commit multiple times and compare results. Mergify Test Engine does that on every PR and on the default branch, scores each test, and surfaces the tests whose pass rate drops below a confidence threshold. Does Cypress retries config fix flaky tests? No, it hides them. A test that fails on attempt 1 and passes on attempt 2 is still broken; you have only decided not to look at the failure. Use Cypress retries config as a temporary bandage for a test you are actively fixing, never as a permanent policy. For visibility without blocking the merge queue, quarantine instead of retry. What causes flaky tests in Cypress? Eight patterns cover most of what we see: cy.wait(ms) racing the page, cy.intercept registered after the request fires, detached DOM errors after re-render, synchronous .then() breaking command-chain retry-ability, session leakage without cy.session(), CSS animations stealing clicks, server state contamination from previous specs, and Cypress retries config hiding real bugs. Each is covered above with a minimal reproducer. How do I quarantine a flaky Cypress test without deleting it? Mergify Test Engine quarantines the test automatically once its confidence score drops. The test still runs in the suite, but a failing result no longer blocks merges and its noise no longer drowns out real signal. When the test stabilizes on main, quarantine lifts automatically. No \`it.skip()\`, no commented-out tests, no orphaned files. Why does my Cypress test pass with cy.wait but fail without it? Because \`cy.wait(ms)\` masked a timing assumption that should have been made explicit. The fix is not to keep the wait, it is to wait for the actual signal: an alias from \`cy.intercept\`, a \`.should()\` assertion that an element exists, or a state-change observable from the DOM. Cypress retries those automatically with the configured timeout, so the spec is fast when the page is fast and patient when CI is slow. ## Ship your Cypress suite green. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/github-actions [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/GitHub Actions # Flaky tests on GitHub Actions. GitHub Actions doesn't know what a test is. It stores, reruns and reports jobs. The tests inside a job are lines in its log. Test Engine adds the missing test layer: a history for every test, flaky detection, quarantine and in-job retries, for every framework you run. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated October 28, 2026 [Start a free 14-day trial](https://dashboard.mergify.com/)[Set up GitHub Actions](https://docs.mergify.com/ci-insights/setup/github-actions) Open a failed run on GitHub Actions and you get a red job and a log. Which test failed, whether it also failed yesterday, whether it fails one run in twenty: GitHub Actions keeps none of it. The unit it stores, reruns and reports is the job. So the one tool it gives you against a flaky test is [Re-run failed jobs](https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs). It starts the failed jobs again on the same commit, every test in them included, and the pull request keeps whichever attempt comes back green. The flaky test is still in the suite, nothing recorded it, and the next pull request pays for it again. Test Engine records every test of every run, on pull requests and on the default branch. A test that passes and fails on the same commit is flagged as flaky. A test you already know is flaky is quarantined: it keeps running and reporting, and its failure stops blocking the merge. On a pull request, the native plugins rerun the new tests inside the job, within a budget you set, so a new flaky test is caught before it lands. Test Engine is the test layer GitHub Actions is missing, and the CI layer is next. Re-run failed jobs GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. ## Pick your framework The GitHub Actions setup for each framework, how its retries compare with rerunning the job, and what makes it flake on GitHub Actions runners. [ pytest on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/pytest)[ Vitest on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/vitest)[ Playwright on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/playwright)[ RSpec on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/rspec)[ Jest on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/jest)[ Go on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/go)[ Cypress on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/cypress)[ JUnit on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/junit)[ TestNG on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/testng)[ Minitest on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/minitest)[ MSTest on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/mstest)[ NUnit on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/nunit)[ Pest on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/pest)[ PHPUnit on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/phpunit)[ Rust on GitHub Actions Read the setup →](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Does GitHub Actions detect flaky tests? No. GitHub Actions records a result per job and keeps the log. It has no notion of an individual test, so it cannot compare one test across runs of the same commit, which is what detecting a flaky test takes. Test Engine records each test's result on every run and flags the ones that both pass and fail on the same code. How do I rerun only the failed tests on GitHub Actions? GitHub Actions cannot: Re-run failed jobs repeats the whole job. Retrying a single test has to happen inside the job, in the test framework. Test Engine's plugins for pytest, Vitest, Playwright, RSpec and Jest rerun new tests on pull requests and known unhealthy tests on the default branch, inside the job that ran them. How do I quarantine a flaky test on GitHub Actions? Without a tool, you skip it in code and open a pull request to do it, then another one to bring it back. With Test Engine, the quarantine list is fetched at the start of the run. A quarantined test still runs and reports, but its failure no longer fails the job, and quarantine lifts when the test is healthy again on the default branch. Does it work on pull requests from forks and from Dependabot? Those runs get no repository secrets, so MERGIFY\_TOKEN is empty. The native plugins then upload nothing and the tests run as usual. With the JUnit upload action, skip the upload step when the secret is missing, as the Mergify docs show. ## Give GitHub Actions a test layer. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/ci-insights/setup/github-actions) --- Source: https://mergify.com/learn/flaky-tests/github-actions/cypress [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/Cypress # Flaky Cypress tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, Cypress reaches Test Engine as JUnit XML. `mocha-junit-reporter` writes one report per spec file, `cypress-multi-reporters` keeps the usual spec output in the log next to it, and the `mergifyio/gha-mergify-ci` action uploads the reports and sets the job's result with quarantine taken into account. Two details decide whether those reports tell the truth: your app has to answer before `cypress run` starts, and Cypress's own retries have to stay off, or a failed attempt is written to the report as a pass. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 5, 2026 Setup ## Run Cypress with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `cypress.config.js`: Point Cypress at cypress-multi-reporters and its config file. Without it, Cypress keeps its default spec reporter and writes no JUnit file. ``` const { defineConfig } = require("cypress"); module.exports = defineConfig({ reporter: "cypress-multi-reporters", reporterOptions: { configFile: "reporter-config.json", }, e2e: { baseUrl: "http://127.0.0.1:8080", }, }); ``` `reporter-config.json`: Keep the spec reporter for the log and write one JUnit file per spec file, named by hash so no spec overwrites another's report. ``` { "reporterEnabled": "spec, mocha-junit-reporter", "mochaJunitReporterReporterOptions": { "mochaFile": "cypress/results/junit-[hash].xml" } } ``` `package.json`: Start the app, wait until it answers with a 200, then run Cypress in the runner's Chrome. Install the tools with npm install --save-dev cypress cypress-multi-reporters mocha-junit-reporter start-server-and-test. ``` { "scripts": { "start": "http-server public -p 8080 -s", "cy:run": "cypress run --browser chrome", "test:cypress": "start-server-and-test start http://127.0.0.1:8080 cy:run" } } ``` `.github/workflows/ci.yml`: Run Cypress with continue-on-error, then upload every report with the Mergify action, which sets the job's result. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest env: HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }} steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 24 cache: npm - run: npm ci - name: Run Cypress tests id: tests continue-on-error: true run: npm run test:cypress - name: Mergify CI Upload if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true' uses: mergifyio/gha-mergify-ci@v28 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: cypress/results/junit*.xml test_step_outcome: ${{ steps.tests.outcome }} - name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true' run: exit 1 ``` - `continue-on-error: true` on the Cypress step is what lets quarantine work: the upload step decides whether the job passes and ignores failures of quarantined tests. `test_step_outcome` tells it when the run itself failed. With one report per spec file, a run that stopped halfway still leaves clean reports for the specs that finished, and those alone would read as green. - Keep `[hash]` in `mochaFile`. Cypress runs each spec file on its own, so a fixed file name is [overwritten by each spec](https://docs.cypress.io/app/tooling/reporters) and only the last spec's results would reach Test Engine. `report_path: cypress/results/junit*.xml` uploads all of them. - Pass `--browser chrome`. Electron is still Cypress's default browser and is [deprecated as a test browser](https://docs.cypress.io/app/references/launching-browsers), and the [Ubuntu runner images](https://docs.cypress.io/app/continuous-integration/github-actions) already ship Chrome, Firefox and Edge. - Pull requests from forks and from Dependabot get no `MERGIFY_TOKEN`. `HAS_MERGIFY_TOKEN` skips the upload on those runs, and the last step fails the job when Cypress failed and nothing was uploaded, since `continue-on-error` would otherwise let those failures through. - Running the suite across browsers in a matrix? Every leg reports the same `GITHUB_JOB`, so give the upload step a `job_name` per leg, such as `job_name: test (${{ matrix.browser }})`. Otherwise a spec that only fails in Firefox looks flaky instead of broken. - `cache: npm` in [setup-node](https://github.com/actions/setup-node) caches npm's package data, and the Cypress binary lives elsewhere, in `~/.cache/Cypress`. So `npm ci` downloads the binary again on every run. [Cypress recommends caching that folder](https://docs.cypress.io/app/continuous-integration/overview); [cypress-io/github-action](https://github.com/cypress-io/github-action) does it for you. Full reference: [Cypress on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/cypress). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs replays the whole job for one spec that may have been unlucky: `npm ci`, the Cypress binary download, the app's start, Chrome's start, then every spec file, the ones that passed included. If you also record to Cypress Cloud, [Cypress's GitHub Actions guide](https://docs.cypress.io/app/continuous-integration/github-actions) lists re-runs that pass with empty tests among its common problems, and passes `GITHUB_TOKEN` to the step so a re-run is identified as one. Cypress's own [test retries](https://docs.cypress.io/app/guides/test-retries) are off by default, `retries: { runMode: 0, openMode: 0 }` per the [configuration reference](https://docs.cypress.io/app/references/configuration). Set `runMode: 2` and a failing test is retried in place up to twice, `beforeEach` and `afterEach` hooks included, and the first attempt that passes is its result. We ran a test that fails on its first attempt only through this setup on GitHub Actions: mocha-junit-reporter wrote it as a plain passing ``, with no trace of the failed attempt. The only evidence left on the runner was a screenshot whose name ends in `(failed).png`. That pass is all Test Engine would receive, so leave `retries` at 0 on the runs you upload. Every failure then reaches Test Engine, a test that fails and passes on the same commit is flagged as flaky, and once it is quarantined, the upload step ignores its failure when it sets the job's result while the spec keeps running and reporting. If you want Cypress to retry anyway, its [experimental](https://docs.cypress.io/app/references/experiments) `detect-flake-but-always-fail` strategy (Cypress 13.4 and later) keeps the failure: in the same experiment, the test came out of the report as failed, with its first attempt's error. Runners ## Why Cypress tests flake on GitHub Actions 1. ### Chrome, Cypress and your app on two vCPUs A [private-repository](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) `ubuntu-latest` runner has 2 vCPUs and 8 GB, where a public repository gets 4 vCPUs and 16 GB. Headless Chrome, the Cypress process and the server you started share them, and most DOM commands still get the same [4-second](https://docs.cypress.io/app/references/configuration) `defaultCommandTimeout` they had on your laptop. The assertion that waits on the slowest response is the first to cross it, on the runs where the runner is busiest. Wait for the request with `cy.intercept` and `cy.wait` where you can; where CI is just slower, [Cypress reads the timeout from the environment](https://docs.cypress.io/app/continuous-integration/overview), so you can raise it for CI alone. ``` - name: Run Cypress tests id: tests continue-on-error: true env: CYPRESS_DEFAULT_COMMAND_TIMEOUT: 8000 run: npm run test:cypress ``` 2. ### An app that is started but not answering Your app runs inside the same job as Cypress, and a server whose process has started may not accept connections yet. `npm start & npx cypress run` is the race [Cypress's CI guide](https://docs.cypress.io/app/continuous-integration/overview) tells you not to write: on a cold runner the first `cy.visit` lands before the port is open, the first spec fails, and the re-run passes because the server won that time. `start-server-and-test`, as in the setup above, waits for a 200 before it starts Cypress; [cypress-io/github-action](https://github.com/cypress-io/github-action) does the same with its `wait-on` input. 3. ### A Chrome version that changes between attempts GitHub updates the browsers in its runner images, and [Cypress notes](https://docs.cypress.io/app/continuous-integration/github-actions) that while a new image is rolling out, the "Set up job" phase can land on the old or the new one. Two attempts on the same commit can then run different Chrome versions, and a failure that only happens on one of them looks flaky. A `cypress/browsers` image with a long-form tag pins Node.js and each browser to an exact version. ``` jobs: test: runs-on: ubuntu-latest # Node.js, Chrome, Firefox and Edge versions fixed by the tag container: image: cypress/browsers:node-24.22.1-chrome-156.0.8078.73-1-ff-159.0-edge-156.0.4314.50-1 ``` 4. ### Spec files split by hand across matrix legs `--parallel` [requires recording to Cypress Cloud](https://docs.cypress.io/app/continuous-integration/overview), so without it, teams split spec files across matrix legs with `--spec`. When each leg starts its own app and database, a spec that relied on data another spec left behind passes while both land on the same leg, and starts failing on the pull request that adds a file and moves it. Keep the leg out of the upload step's `job_name`: the legs run different specs, and a name per leg would split a spec's history each time it moves. The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which Cypress test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [Cypress guide](https://mergify.com/learn/flaky-tests/cypress) for the patterns behind flaky Cypress suites on any CI: cy.wait() races, network stubbing, and retry-ability gotchas. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my Cypress tests pass locally but fail on GitHub Actions? On GitHub Actions, cypress run is headless, on 2 vCPUs shared with your app when the repository is private, against a server that may not be answering yet, in whichever Chrome the runner image ships. Locally you likely run cypress open, headed, on a faster machine. Reproduce with npx cypress run --browser chrome, headless like CI, against a server started the way CI starts it, and add --headed --no-exit to watch the run and inspect the page once it ends. Should I turn on retries in cypress.config.js? Not on the runs you upload. A test that fails and then passes on its retry is written to the JUnit report as a plain pass, so Test Engine never sees the failure and cannot flag it. Leave retries at 0 and let quarantine keep the known flaky specs from blocking merges. If you keep retries, the experimental detect-flake-but-always-fail strategy reports such a test as failed. Can I keep cypress-io/github-action? Yes. Give the action step id: tests and continue-on-error: true, and keep the Mergify upload step after it. The reporters are set in cypress.config.js, so they apply however Cypress is started, and the action's start and wait-on inputs replace start-server-and-test. It also caches the Cypress binary, which setup-node does not. ## Catch the flaky Cypress tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/cypress) --- Source: https://mergify.com/learn/flaky-tests/github-actions/go [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/Go # Flaky Go tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, Go tests reach Test Engine as a JUnit report: `gotestsum` writes it, and the `mergifyio/gha-mergify-ci` action uploads it and decides the job's result with quarantine taken into account. The detail that matters most on GitHub Actions is the test cache, which can skip the very tests you are trying to watch. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 5, 2026 Setup ## Run Go with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `.github/workflows/ci.yml`: Write junit.xml with gotestsum, then upload it with the Mergify action, which sets the job's result. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest env: HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }} steps: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: go-version: stable - name: Run tests id: tests continue-on-error: true run: go run gotest.tools/gotestsum@v1.13.0 --junitfile junit.xml -- -count=1 ./... - name: Mergify CI Upload if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true' uses: mergifyio/gha-mergify-ci@v27 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: junit.xml test_step_outcome: ${{ steps.tests.outcome }} - name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true' run: exit 1 ``` - `continue-on-error: true` on the test step is what lets quarantine work: the upload step decides whether the job passes, ignoring failures of quarantined tests. `test_step_outcome` tells it when the test run itself crashed, so a panic that left a clean report does not pass for green. - Pull requests from forks and from Dependabot get no `MERGIFY_TOKEN`. `HAS_MERGIFY_TOKEN` skips the upload there, and the last step fails the job when tests failed and nothing was uploaded, or `continue-on-error` would let those failures through. - Recent `gotestsum` releases need a recent Go (v1.13.0 requires Go 1.24). Pin an older release if your module targets an older Go. - Feed the converter `go test`'s standard output only. Merging stderr turns linker warnings into failing tests. Full reference: [Go on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/golang). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs recompiles every package and reruns every test for one that failed, and with the default `setup-go` cache it may not even rerun the test you care about (see below). Go has no retry of its own. `gotestsum --rerun-fails` adds one, and like every blanket retry it reports the passing attempt, so the flake is absorbed. With Test Engine, every upload adds to each test's history on the default branch and on pull requests, which is how a test that passes and fails on the same commit gets flagged. Once a flaky test is quarantined, the upload step ignores its failure when it sets the job's result, and the test keeps running and reporting until it is healthy again. Runners ## Why Go tests flake on GitHub Actions 1. ### The test cache skips the flaky test `actions/setup-go` caches the build cache by default, and `go test` reuses cached results for packages whose inputs did not change. A flaky test in an untouched package prints `(cached)` and does not run at all, so it cannot fail, and it cannot report either. `-count=1` turns result caching off, which is what you want on the runs that feed flaky detection. 2. ### \-race on two cores The race detector multiplies execution time by 2 to 20 and memory by 5 to 10, per the Go documentation. On a [private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) with 2 vCPUs and 8 GB, deadlines written as `time.After(100 * time.Millisecond)` that hold on a laptop start to miss. Wait on a condition, not on a duration. ``` // testify: wait for the condition, not for a fixed delay require.Eventually(t, func() bool { return srv.Ready() }, 5*time.Second, 10*time.Millisecond) ``` 3. ### t.Parallel() at the runner's width `-parallel` defaults to `GOMAXPROCS`, which is the runner's CPU count: 2 on a private runner, 4 on a public one, often 10 or more on a laptop. Parallel tests that share a port, a temp directory or a package-level variable interleave differently on each, so a race that never shows locally shows on CI, and the other way around. The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which Go test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [Go guide](https://mergify.com/learn/flaky-tests/go) for the patterns behind flaky Go suites on any CI: t.Parallel() races, goroutine leaks, and nondeterministic map iteration. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my Go tests pass locally and fail on GitHub Actions? Usually the runner's width and speed: GOMAXPROCS and -parallel follow its 2 or 4 vCPUs, and -race slows it down further. Reproduce with GOMAXPROCS=2 go test -race -count=1 on the failing package. Why does a flaky Go test never show up in my reports? If the package did not change, go test can serve its result from the cache restored by setup-go and print (cached) instead of running it. Run with -count=1 so every test runs and reports. Why does the upload step need test\_step\_outcome? With continue-on-error on the test step, the upload step decides the job's result. test\_step\_outcome tells it that go test failed, so a crash that left a partial but clean junit.xml fails the job instead of passing it. ## Catch the flaky Go tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/golang) --- Source: https://mergify.com/learn/flaky-tests/github-actions/jest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/Jest # Flaky Jest tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, Jest needs one package, one wrapper and one secret: install `@mergifyio/jest`, wrap your config with `withMergify`, and give the test step a `MERGIFY_TOKEN`. The reporter uploads every result when the run ends and applies quarantine inside Jest's workers, with no JUnit file to write or upload. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated October 28, 2026 Setup ## Run Jest with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `jest.config.js`: Wrap the Jest config with withMergify, which adds the reporter and test runner. ``` const { withMergify } = require("@mergifyio/jest"); module.exports = withMergify({ testEnvironment: "node", // ...your existing configuration }); ``` `.github/workflows/ci.yml`: Pass the MERGIFY\_TOKEN secret to the step that runs Jest. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 24 cache: npm - run: npm ci - name: Run tests env: MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }} run: npm test # jest ``` - Remove `--reporters` from the test command. On the command line it replaces the reporters of your config, the Mergify one included. - Config in `package.json`? `withMergify` needs a function call, so move it to `jest.config.js` first. - Moving from `jest-junit` and the upload action? Drop the upload step and `continue-on-error` from the test step: the reporter sends the results and quarantine is already in Jest's exit code. - Running Jest across a matrix? Every leg reports the same `GITHUB_JOB`, so set `MERGIFY_TEST_JOB_NAME: test (${{ matrix.node-version }})` on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken. Full reference: [Jest on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/jest). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs repeats `npm ci` and every test file for one test that may have been unlucky. `jest.retryTimes(2)` retries inside the run and reports the test as passed when an attempt succeeds. The flake is still there, and only the log of that run shows the failed attempts. `@mergifyio/jest` turns Jest's own retries off during its reruns and decides with history instead. On a pull request it reruns the tests the target branch does not have yet, in Jest's workers, within the Prevention budget, and fails the run if any attempt fails. On the default branch it reruns the tests Test Engine knows as unhealthy without changing the result. Runners ## Why Jest tests flake on GitHub Actions 1. ### One worker on a private-repository runner Jest's `maxWorkers` defaults to the number of CPUs minus one. On a [2-vCPU private runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) that is a single worker, so files that ran in parallel on your laptop run one after the other, and a slow file can push tests past the default 5-second timeout. On a public repository's 4 vCPUs it is three workers. Reproduce with the runner's count. ``` npx jest --maxWorkers=1 ``` 2. ### Workers running out of memory A private runner has 8 GB for the runner, your app's services and every Jest worker. Workers that leak memory across files get killed mid-file, and the failure lands on whichever test was running. `workerIdleMemoryLimit` recycles a worker before it gets there. ``` module.exports = withMergify({ workerIdleMemoryLimit: "512MB", }); ``` 3. ### Snapshots that only exist on one machine Jest runs with `--ci` by default on GitHub Actions, where a missing snapshot fails the test instead of being written. A test whose snapshot was never committed passes locally and fails on CI, and goes "green" the moment someone commits it. It looks flaky; it is a missing file. The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which Jest test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [Jest guide](https://mergify.com/learn/flaky-tests/jest) for the patterns behind flaky Jest suites on any CI: fake timers, snapshot drift, and the async traps behind flaky Jest suites. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why are my Jest tests flaky only on GitHub Actions? Look at the runner first: one worker on a 2-vCPU private runner instead of one per core on your laptop, 8 GB shared by every worker, and a UTC clock. Run with --maxWorkers=1 and TZ=UTC locally to reproduce. Do I still need jest-junit? Not for Test Engine: @mergifyio/jest uploads the results itself. If another tool reads the JUnit file, list jest-junit in the config's reporters, after which withMergify adds its own. Are test.concurrent tests rerun? No. jest-circus starts them together with the other tests of their file and cannot run one again on its own. They are still reported and can still be quarantined. ## Catch the flaky Jest tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/jest) --- Source: https://mergify.com/learn/flaky-tests/github-actions/junit [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/JUnit # Flaky JUnit tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, JUnit results reach Test Engine through the XML reports Maven Surefire already writes, one `TEST-*.xml` per test class, with nothing to add to the pom. The `mergifyio/gha-mergify-ci` action uploads the reports of every module and decides the job's result with quarantine taken into account. The setting to watch is Surefire's own retry: with `rerunFailingTestsCount` on, a test that fails and then passes is uploaded as a pass. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 5, 2026 Setup ## Run JUnit with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `pom.xml`: A plain JUnit 5 project. Surefire runs the tests and writes one JUnit XML report per test class on its own, so nothing Mergify-specific goes in the pom. ``` 4.0.0 com.example te-demo-junit 1.0-SNAPSHOT 21 UTF-8 org.junit junit-bom 5.14.4 pom import org.junit.jupiter junit-jupiter test org.apache.maven.plugins maven-surefire-plugin 3.6.1 ``` `.github/workflows/ci.yml`: Run Maven with continue-on-error, then upload every module's Surefire reports with the Mergify action, which sets the job's result. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest env: HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }} steps: - uses: actions/checkout@v7 - uses: actions/setup-java@v6 with: distribution: temurin java-version: 21 cache: maven - name: Run tests id: tests continue-on-error: true run: mvn -B -fae test - name: Mergify CI Upload if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true' uses: mergifyio/gha-mergify-ci@v28 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: "**/target/surefire-reports/TEST-*.xml" test_step_outcome: ${{ steps.tests.outcome }} - name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true' run: exit 1 ``` - `continue-on-error: true` on the Maven step hands the job's result to the upload step, which leaves out failures of quarantined tests. `test_step_outcome` passes Maven's own result along, so a build that failed for a reason the reports do not show, such as a test class that does not compile, still fails the job. - `-fae` keeps Maven going after a module's tests fail, so every module runs its tests and writes its reports, and `**/target/surefire-reports/TEST-*.xml` picks them all up. Integration tests on Failsafe? Run `mvn -B -fae verify` and upload `**/target/failsafe-reports/TEST-*.xml` as well. Keep the `TEST-` prefix: Failsafe also writes a `failsafe-summary.xml` in that directory, which is not a JUnit report, and an upload that includes it fails. - Pull requests from forks and from Dependabot run without `MERGIFY_TOKEN`. `HAS_MERGIFY_TOKEN` skips the upload there, and the last step fails the job when tests failed and nothing was uploaded, since `continue-on-error` would otherwise let those failures through. - Testing on several JDKs in a matrix? Every leg reports the same `GITHUB_JOB`, so give the upload step a `job_name` per leg, such as `job_name: test (${{ matrix.java }})`. Otherwise a test that only fails on Java 17 looks flaky instead of broken. - On Gradle, run `./gradlew test --continue` and upload `**/build/test-results/**/*.xml`. Gradle [names tests differently](https://docs.mergify.com/test-engine/test-frameworks/junit#test-names) from Maven (`addsAnItem()` where Maven reports `addsAnItem`), so moving a project from one build tool to the other starts new histories in Test Engine. Full reference: [JUnit on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/junit). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs starts the job from the top. The `cache: maven` input of [setup-java](https://github.com/actions/setup-java) brings `~/.m2/repository` back, so dependencies are not downloaded again, but the build itself starts over: Maven compiles every module and Surefire runs every test class, to retry the one test that failed. The pull request keeps whichever attempt comes back green. Surefire's own [rerunFailingTestsCount](https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html) retries inside the run, right after a test fails. Its [default is 0](https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html), which turns it off. We ran a test that fails 40% of the time through the setup above, with `-Dsurefire.rerunFailingTestsCount=2` added to the Maven command. It failed, passed on its rerun, and Maven ended with `BUILD SUCCESS` and a `Flakes: 1` warning. The report did keep the failure, as a `` element inside a passing ``, but the upload step counted 4 tests and 0 failures, and Test Engine stored a pass. So leave `rerunFailingTestsCount` at 0 on the runs you upload. Every failure then reaches Test Engine, which records each test's result on pull requests and on the default branch, and a test that fails and passes on the same commit is flagged as flaky. Once it is quarantined, the upload step leaves its failure out of the job's result, and the test keeps running and reporting until it is healthy again. Runners ## Why JUnit tests flake on GitHub Actions 1. ### JUnit's parallel mode sized by the runner Run JUnit Jupiter tests in parallel without choosing a strategy, and it [uses the dynamic one with a factor of 1](https://docs.junit.org/current/writing-tests/parallel-execution.html): the parallelism equals the number of available processors. That is 2 on a [private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) and 4 on a public one, whatever your laptop has. Tests that share a static field or a database row meet in a different order at each width, so a race that never shows on your machine shows on CI. Pin the parallelism to the runner's to reproduce it. ``` # src/test/resources/junit-platform.properties junit.jupiter.execution.parallel.enabled = true junit.jupiter.execution.parallel.mode.default = concurrent junit.jupiter.execution.parallel.config.strategy = fixed junit.jupiter.execution.parallel.config.fixed.parallelism = 2 ``` 2. ### Test classes in the order the disk lists them By default, Surefire runs every test class in one JVM: [forkCount is 1 and reuseForks is true](https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html), so a static field one class sets is still set when the next class starts. Which class comes next depends on `runOrder`, whose default is `filesystem`, the order in which the class files are listed on disk. Nothing makes that order match between your laptop and an Ubuntu runner, so a class that only passes after another one can start failing on CI. `random` shuffles the classes and prints the seed to replay the order; since Surefire 3.6.0 it also shuffles the methods inside each JUnit 5 class. ``` # Shuffle, then replay the order of a failing run with the seed Surefire printed mvn -B test -Dsurefire.runOrder=random mvn -B test -Dsurefire.runOrder=random -Dsurefire.runOrder.random.seed= ``` 3. ### A hung test that holds the job for six hours A test that never returns, stuck on a lock or waiting for a container that never comes up, keeps the job running until GitHub cancels it, [after 360 minutes](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idtimeout-minutes) by default. JUnit Jupiter reads a [default timeout](https://docs.junit.org/current/writing-tests/timeouts.html) for every test and lifecycle method from [junit-platform.properties](https://docs.junit.org/current/running-tests/configuration-parameters.html), and can dump every thread's stack to standard output when one runs out. The hung test then fails in minutes, the failure reaches Test Engine like any other, and the dump shows where it was stuck. The default thread mode interrupts the test's thread, so code that swallows the interrupt keeps going; Surefire's [`forkedProcessTimeoutInSeconds`](https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html) is the backstop that kills the test JVM. ``` # src/test/resources/junit-platform.properties junit.jupiter.execution.timeout.default = 5 m junit.jupiter.execution.timeout.threaddump.enabled = true ``` The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which JUnit test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [JUnit guide](https://mergify.com/learn/flaky-tests/junit) for the patterns behind flaky JUnit suites on any CI: method ordering, Surefire forking, and shared static state in Spring tests. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my JUnit tests pass locally and fail on GitHub Actions? Start with the runner and the order. A private repository's runner has 2 vCPUs, so tests you run in parallel go 2 at a time there under JUnit's default strategy, and Surefire runs your test classes one after the other in a single JVM, in whatever order the runner's disk lists them. Reproduce with a fixed parallelism of 2 in junit-platform.properties and mvn test -Dsurefire.runOrder=random, then replay a failing order with the seed Surefire prints. Should I use Surefire's rerunFailingTestsCount? Not on the runs you upload. A test that fails and then passes on its rerun is written as a passing testcase with a flakyFailure element inside, and the upload counts it as a pass, so Test Engine never sees the failure it needs to flag the test. Leave it at 0 and let quarantine keep the known flaky tests from blocking merges. Does this work with Gradle instead of Maven? Yes. Run ./gradlew test --continue, so every project runs its tests even after one fails, and upload \*\*/build/test-results/\*\*/\*.xml. Gradle names tests differently from Maven (a method name ends in parentheses, for one), so a project that moves from Maven to Gradle starts new test histories in Test Engine. ## Catch the flaky JUnit tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/junit) --- Source: https://mergify.com/learn/flaky-tests/github-actions/minitest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/Minitest # Flaky Minitest tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, Minitest needs a gem, one require and one secret: add `minitest-mergify` to the test group, require `minitest/mergify` from your test helper, and give the step that runs the tests a `MERGIFY_TOKEN`. The plugin uploads every test's result when the run ends, applies quarantine inside Minitest, and reruns new tests before the pull request can merge. Rails test suites run on Minitest, so the same setup covers `bin/rails test`. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 5, 2026 Setup ## Run Minitest with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `Gemfile`: Add minitest-mergify to the test group of the Gemfile. In a Rails app, minitest is already there. ``` source 'https://rubygems.org' group :test do gem 'minitest' gem 'minitest-mergify' gem 'rake' end ``` `test/test_helper.rb`: Require the plugin in the test helper, before any test is defined. Minitest 6 does not load it otherwise. ``` require 'minitest/autorun' require 'minitest/mergify' require 'calc' ``` `.github/workflows/ci.yml`: Pass the MERGIFY\_TOKEN secret to the step that runs Minitest. In a Rails app, keep bin/rails test. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: ruby/setup-ruby@v1 with: ruby-version: "3.4" bundler-cache: true - name: Run tests env: MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }} run: bundle exec rake test ``` - Minitest 6 loads only the plugins a suite asks for, so without `require 'minitest/mergify'` the run goes as usual and nothing reaches Test Engine. Keep the line on Minitest 5 too: there the plugin would load only after the tests are defined, and spec tests with no body lose their line number. - No `continue-on-error` and no `test_step_outcome`: there is no upload step to hand the result to. A quarantined test that fails counts as a skip, so the step's exit code is already the job's result. A test file that fails to load still fails the step. - Running Minitest across a matrix? Every leg reports the same `GITHUB_JOB`, so set `MERGIFY_TEST_JOB_NAME: test (${{ matrix.ruby-version }})` on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken. - CRuby 3.1 to 4.0, with Minitest 5.19 or later, or Minitest 6 on Ruby 3.2 and later. A JRuby or `ruby-head` leg runs its tests but reports nothing; give that leg the JUnit reporter from `minitest-reporters` and the `mergifyio/gha-mergify-ci` upload step instead. - The gem runs on `mergify-ci-core`, precompiled per platform. If your `Gemfile.lock` was written on a Mac, run `bundle lock --add-platform x86_64-linux` and commit it, so Bundler can install the Linux build on the runner. - [bin/rails test leaves system tests out](https://guides.rubyonrails.org/testing.html#running-tests-in-continuous-integration-ci). Run `bin/rails test:all` in the step if you want them reported too. Full reference: [Minitest on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/minitest). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs replays the job from the top: setup-ruby, the bundle restored from cache, the database setup if you have one, then the whole suite. Minitest also picks a new seed for that attempt (the log prints it as `Run options: --seed`), so the attempt that passes did not even run the tests in the same order as the one that failed. [minitest-retry](https://github.com/y-yagi/minitest-retry) moves the retry inside the run: a failing test is retried up to 3 times by default, and it counts as passed if one attempt passes. The `[MinitestRetry] retry` lines in that job's log are the only trace of the flake. minitest-mergify reruns with a narrower rule. On a pull request, once the suite is over, it reruns the tests the target branch does not have yet, within the Prevention budget, through Minitest itself, so a test that ran in Rails' parallel workers is rerun there. If one attempt fails, the test is flaky and the job fails. On pushes to the default branch it reruns the tests Test Engine knows as unhealthy, within the Detection budget, without changing the result. Runners ## Why Minitest tests flake on GitHub Actions 1. ### A worker count that follows the runner `parallelize(workers: :number_of_processors)` forks one worker per core, which is [2 on a private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) and 4 on a public one, against often 10 or more on a laptop. [Rails also skips parallelization below 50 tests](https://guides.rubyonrails.org/testing.html#parallel-testing), so the test file you run locally usually runs in one process, while CI runs the full suite across workers, each with its own share of the tests. Reproduce with the runner's worker count and the seed from the failed attempt's log. ``` PARALLEL_WORKERS=2 bin/rails test --seed 39840 ``` 2. ### Workers that share everything but the database [Active Record gives each worker its own database](https://guides.rubyonrails.org/testing.html#parallel-testing-with-processes), suffixed with the worker's number (`test-database-0`, `test-database-1`). The rest is yours to split. Active Storage's Disk service folder or a Redis service container is shared by every worker of the job, so one worker's teardown can delete the files another worker's test is still reading. [The Active Storage guide](https://guides.rubyonrails.org/active_storage_overview.html#discarding-files-created-during-tests) gives each worker its own folder in `parallelize_setup`. ``` class ActionDispatch::IntegrationTest parallelize_setup do |i| ActiveStorage::Blob.service.root = "#{ActiveStorage::Blob.service.root}-#{i}" end end ``` 3. ### A new seed on every attempt [Test methods run in random order](https://guides.rubyonrails.org/testing.html), and Minitest prints the seed before the first one. Every attempt of the job draws a new seed, so a test that depends on what ran before it fails on one attempt and passes on the next, and Re-run failed jobs reads that as a fluke. Rerun with the failed attempt's seed (`--seed`, or `SEED=` with Rake) before calling the test flaky. ``` SEED=39840 bundle exec rake test ``` The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which Minitest test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [Minitest guide](https://mergify.com/learn/flaky-tests/minitest) for the patterns behind flaky Minitest suites on any CI: parallelization races, fixture loading, and test isolation in Rails suites. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my Rails tests pass locally and fail on GitHub Actions? Usually the runner, not the test: 2 parallel workers on a private repository, a new seed on every attempt, and a full suite that runs in parallel while the one file you ran locally did not (Rails only parallelizes 50 tests or more). Run the suite with PARALLEL\_WORKERS=2 and the seed printed in the failed attempt's log. Why does Minitest report 0 failures when the job failed? On a pull request, minitest-mergify reruns new tests after the suite. Minitest's summary and your other reporters only show your own run, so a new test that failed on a rerun still counts as passed there. The exit code and the plugin's # Mergify CI: section in the log say it is flaky. Do I still need minitest-reporters or minitest-junit on GitHub Actions? Not for Test Engine: minitest-mergify uploads the results itself. Keep the JUnit reporter only for a leg the plugin cannot report from, such as JRuby. Moving from a JUnit upload keeps your quarantine entries, while the tests' history starts again under their new names. Does minitest-mergify work with Rails parallel tests? Yes, whether the tests run in Minitest's threads (parallelize\_me!) or in Rails' parallel workers, and its reruns go through those workers too. Quarantine applies the same way as in a single process. ## Catch the flaky Minitest tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/minitest) --- Source: https://mergify.com/learn/flaky-tests/github-actions/mstest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/MSTest # Flaky MSTest tests on GitHub Actions. Set up, retried right, quarantined. MSTest reaches Test Engine on GitHub Actions through a JUnit report. `JunitXml.TestLogger` writes `junit.xml` during `dotnet test`, and the `mergifyio/gha-mergify-ci` action uploads it, then sets the job's result with the failures of quarantined tests left out. Before anything else, check which test platform the project runs on: on `MSTest.Sdk`, `dotnet test` runs Microsoft.Testing.Platform, which drops `--logger` without an error and leaves the upload step nothing to send. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 6, 2026 Setup ## Run MSTest with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `Demo.Tests.csproj`: Reference JunitXml.TestLogger next to the MSTest package, on Microsoft.NET.Sdk, where dotnet test runs on VSTest. ``` net10.0 enable enable false ``` `.github/workflows/ci.yml`: Run dotnet test with the JUnit logger, then let the Mergify action upload junit.xml and decide the job's result. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest env: HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }} steps: - uses: actions/checkout@v7 - uses: actions/setup-dotnet@v6 with: dotnet-version: 10.0.x - name: Run tests id: tests continue-on-error: true run: dotnet test --logger "junit;LogFilePath=junit.xml" - name: Mergify CI Upload if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true' uses: mergifyio/gha-mergify-ci@v28 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: junit.xml test_step_outcome: ${{ steps.tests.outcome }} - name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true' run: exit 1 ``` - `MSTest.Sdk` sets `EnableMSTestRunner` and `TestingPlatformDotnetTestSupport` to true, so its tests [run on Microsoft.Testing.Platform](https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-sdk), under `dotnet test` too. In that mode, VSTest options such as `--logger` are [silently ignored](https://learn.microsoft.com/dotnet/core/testing/unit-testing-with-dotnet-test): the tests pass or fail as usual and `junit.xml` is never written. Add `true` to the project to switch it back. The project above uses the `MSTest` package on `Microsoft.NET.Sdk`, which stays on VSTest as long as nothing sets `TestingPlatformDotnetTestSupport`. - The logger resolves `LogFilePath` from the test project's folder. With the `.csproj` at the repository root, as here, `report_path: junit.xml` finds the file. For a project in `tests/Shop.Tests/`, upload `tests/Shop.Tests/junit.xml`. - The upload step owns the job's result. `continue-on-error: true` stops a red test step from failing the job by itself, and the upload step fails it unless every failing test is quarantined. `test_step_outcome` passes it the test step's real outcome, so a test host that crashed after writing part of the report still fails the job, even when that report shows no failure. - `MERGIFY_TOKEN` is empty on pull requests from forks and from Dependabot. `HAS_MERGIFY_TOKEN` skips the upload on those runs, and the last step fails the job when the tests failed, which `continue-on-error` would otherwise let through. - A `--filter` that matches no test still [exits 0](https://learn.microsoft.com/dotnet/core/tools/dotnet-test-vstest#exit-codes) with a warning, and the upload step receives a report with no test in it. Set `RunConfiguration.TreatNoTestsAsError` to true in a `.runsettings` file passed with `--settings`, so a typo in a filter fails the job instead of passing it with nothing tested. - Running the suite on `ubuntu-latest` and `windows-latest` from one matrix? Both legs report the same `GITHUB_JOB`, so add `job_name: test (${{ matrix.os }})` to the upload step. Without it the two legs share one history per test, and a test that only fails on Windows looks flaky instead of broken on Windows. Full reference: [MSTest on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/mstest). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. [Re-run failed jobs](https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs) starts the test job again on the same commit: setup-dotnet, the NuGet restore, the build, then every test in the assembly, to find out whether one method was unlucky. The pull request shows the attempt that comes back green. The red one stays in the run's history, which nobody opens once the check has passed. MSTest has had its own [\[Retry\]](https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-writing-tests-controlling-execution#retry-attributes) since 3.8, on a test method or, from MSTest 4.3, on a whole class. `[Retry(3)]` runs a test that fails or times out up to three more times, with an optional delay and exponential backoff between attempts. On VSTest, where the JUnit logger runs, only the final result is reported: a test that failed once and then passed reaches `junit.xml` as a plain pass, and the failure is gone from every report. Test Engine works from the failures `[Retry]` would have hidden. Every upload adds each test's result to its history, on pull requests and on the default branch, and a test that fails and passes on the same commit is flagged as flaky. Take `[Retry]` off the tests you want watched and quarantine the ones you already know are flaky: the upload step leaves a quarantined test's failure out of the job's result, and the test keeps running and reporting until it is healthy again. Runners ## Why MSTest tests flake on GitHub Actions 1. ### Method-level parallelism at the runner's width `[assembly: Parallelize]` without a `Workers` value runs [one worker per logical processor](https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-writing-tests-controlling-execution#worker-threads). A [private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) gives it 2 workers and a public one 4, usually fewer than your laptop. With `Scope = ExecutionScope.MethodLevel`, any two test methods can overlap, in the same class or not, and which ones do depends on the worker count. Two tests that write the same database row may never overlap on your machine and collide on CI, or the reverse. Pin the width to reproduce the runner. On MSTest 4.4 and later, give the tests that share one resource the same [\[ResourceLock\]](https://learn.microsoft.com/dotnet/api/microsoft.visualstudio.testtools.unittesting.resourcelockattribute) key: MSTest then runs those tests one at a time and keeps the rest of the suite parallel. ``` // Reproduce a private-repository runner's width locally [assembly: Parallelize(Workers = 2, Scope = ExecutionScope.MethodLevel)] ``` 2. ### A \[Timeout\] that fires on CI and keeps running `[Timeout(2000)]` is [in milliseconds](https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-writing-tests-controlling-execution#timeout-attributes), and a budget that fits a laptop can be too short on a [2-vCPU runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners). What happens after the timeout does more damage than the failure itself. By default, MSTest runs a timed test in a separate task, and when time is up it stops watching that task without stopping it. The test goes on writing to whatever it shares with the tests that run after it, and the next failure lands on a test that did nothing wrong. Since MSTest 3.6, `CooperativeCancellation = true` cancels a token instead. Pass that token all the way down, so a timed-out test actually stops. ``` [TestMethod] [Timeout(10_000, CooperativeCancellation = true)] public async Task ImportsOrders(CancellationToken cancellationToken) { await importer.RunAsync(cancellationToken); } ``` 3. ### A shuffled order that changes on every attempt MSTest 4.3 added `RandomizeTestOrder`, which [runs tests in a random order](https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-configure#mstest-element) to bring out tests that depend on what ran before them. Without `RandomTestOrderSeed`, each run draws a new order, so the attempt you start with Re-run failed jobs runs a different sequence and can pass for reasons nobody sees. Seed it with `github.run_number`, which GitHub [keeps on a re-run](https://docs.github.com/en/actions/reference/workflows-and-actions/contexts) and increments for each new run. Every attempt of a run then replays one order, and the run number is all you need to replay that order locally. ``` - name: Run tests id: tests continue-on-error: true run: dotnet test --logger "junit;LogFilePath=junit.xml" -- MSTest.RandomizeTestOrder=true MSTest.RandomTestOrderSeed=${{ github.run_number }} ``` The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which MSTest test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [MSTest guide](https://mergify.com/learn/flaky-tests/mstest) for the patterns behind flaky MSTest suites on any CI: assembly initialization, DeploymentItem races, and parallel test settings. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my MSTest tests pass locally and fail on GitHub Actions? Start with the worker count. \[assembly: Parallelize\] uses one worker per logical processor, which is 2 on a private repository's runner, so method-level tests overlap in different pairs than on your laptop. Then look at \[Timeout\] values in milliseconds that were tuned on a faster machine. Pin Workers = 2 and run the failing class locally before you call a test flaky. Why does my MSTest.Sdk project write no junit.xml? MSTest.Sdk runs dotnet test on Microsoft.Testing.Platform, which drops VSTest options like --logger without failing the run. From MTP 1.9 the log carries an MTP0001 warning for it. Set UseVSTest to true in the project file and the JUnit logger runs again. If the file is still missing, look in the test project's folder: LogFilePath is relative to the .csproj, not to the repository root. Does MSTest's \[Retry\] attribute hide flaky tests? On VSTest it does. Only the last attempt of a retried test is reported, so a test that failed and then passed reaches junit.xml as passed, and Test Engine never sees the failure. Keep \[Retry\] off the tests you want watched and quarantine the known flaky ones instead: they keep running and reporting, and their failures stop failing the job. Can I skip a flaky test on CI with \[CICondition\]? You can: \[CICondition(ConditionMode.Exclude)\], in MSTest 3.10 and later, skips the test whenever GITHUB\_ACTIONS or CI is true, and GitHub Actions sets both on every run. The test then never runs on CI, so nothing shows when it is fixed and nothing catches a regression in the code it covers. Quarantine keeps it running and reporting while its failures stop counting. ## Catch the flaky MSTest tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/mstest) --- Source: https://mergify.com/learn/flaky-tests/github-actions/nunit [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/NUnit # Flaky NUnit tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, NUnit results reach Test Engine as a JUnit report. Add `JunitXml.TestLogger` to the test project so `dotnet test` writes `junit.xml`, then upload it with the `mergifyio/gha-mergify-ci` action, which also decides whether the job passes once quarantined tests are set aside. The detail to get right is where the report lands: the logger resolves its path from the test project's folder, not from the repository root. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 5, 2026 Setup ## Run NUnit with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `Demo.Tests.csproj`: Add JunitXml.TestLogger to the test project, next to NUnit and its test adapter. ``` net10.0 enable enable false ``` `.github/workflows/ci.yml`: Write junit.xml with the JUnit logger, then upload it with the Mergify action, which sets the job's result. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest env: HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }} steps: - uses: actions/checkout@v7 - uses: actions/setup-dotnet@v6 with: dotnet-version: 10.0.x - name: Run tests id: tests continue-on-error: true run: dotnet test --logger "junit;LogFilePath=junit.xml" - name: Mergify CI Upload if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true' uses: mergifyio/gha-mergify-ci@v28 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: junit.xml test_step_outcome: ${{ steps.tests.outcome }} - name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true' run: exit 1 ``` - `LogFilePath` is resolved from the folder that holds the test project's `.csproj`, not from the repository root. The project above sits at the root, so `report_path: junit.xml` matches. For a project in `tests/Api.Tests/`, upload `tests/Api.Tests/junit.xml`. - A solution with several test projects writes one report per project, all with the same name. Give the logger an absolute path with the [{assembly} token](https://github.com/spekt/testlogger/wiki/Logger-Configuration), `--logger "junit;LogFilePath=${{ github.workspace }}/test-results/{assembly}.xml"`, and upload `test-results/*.xml`. - `continue-on-error: true` hands the job's result to the upload step, which leaves out failures of quarantined tests. `dotnet test` [exits 1](https://learn.microsoft.com/dotnet/core/tools/dotnet-test-vstest) for a failing test, and also when a test assembly cannot load or the run aborts. `test_step_outcome` makes sure that second kind fails the job even when the report it left behind looks clean. - Pull requests from forks and from Dependabot run without `MERGIFY_TOKEN`. `HAS_MERGIFY_TOKEN` skips the upload there, and the last step fails the job when tests failed and nothing was uploaded. - Turned on `EnableNUnitRunner`? On its own it changes nothing for `dotnet test`. Add `TestingPlatformDotnetTestSupport` and the tests run on Microsoft.Testing.Platform, which [silently ignores](https://learn.microsoft.com/dotnet/core/testing/unit-testing-with-dotnet-test) VSTest options such as `--logger`: the suite runs, and no `junit.xml` appears. Full reference: [NUnit on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/nunit). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs starts again from the NuGet restore and the build, then runs every fixture to retry the one test that failed. The pull request keeps whichever attempt comes back green. NUnit's [\[Retry(3)\]](https://docs.nunit.org/articles/nunit/writing-tests/attributes/retry.html) retries inside the run instead. The number is the total count of attempts, and only an assertion failure triggers another one: a test that throws an unexpected exception is reported as an error and not retried, unless that exception type is listed in `RetryExceptions` (NUnit 4.5 and later). When a later attempt passes, the test counts as passed and the race stays in the code. Test Engine records every run instead of keeping the lucky one. Each upload adds to each test's history, on pull requests and on the default branch, so a test that fails and passes on the same commit is flagged as flaky. Once it is quarantined, the upload step leaves its failure out of the job's result, and the test keeps running and reporting until it is healthy again. Runners ## Why NUnit tests flake on GitHub Actions 1. ### Two workers on a private-repository runner Unless the assembly sets `[LevelOfParallelism]`, NUnit runs `[Parallelizable]` tests on [as many workers as the machine has processors, or 2, whichever is greater](https://docs.nunit.org/articles/nunit/writing-tests/attributes/levelofparallelism.html). That is 2 on a [private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) and 4 on a public one, whatever your laptop has. Fixtures that share a static field or a database row interleave differently at each width, so a race that never shows on your machine shows on CI, and the other way around. Reproduce with the runner's worker count, which the NUnit adapter [reads from the run settings](https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html). ``` dotnet test -- NUnit.NumberOfTestWorkers=2 ``` 2. ### A hung test that holds the job for six hours A test that deadlocks, typically on `.Result` for a task that needs the thread it is blocking, keeps the job running until GitHub cancels it, [after 360 minutes](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idtimeout-minutes) by default. Once the job is cancelled, [neither success() nor failure()](https://docs.github.com/en/actions/reference/workflows-and-actions/expressions) is true, so the upload step never runs and the hang leaves no result in Test Engine. `--blame-hang-timeout` has VSTest [dump and stop the test host](https://learn.microsoft.com/dotnet/core/tools/dotnet-test-vstest) when one test runs past the limit, so the test step fails in minutes and the dump shows where it was stuck. ``` dotnet test --logger "junit;LogFilePath=junit.xml" --blame-hang-timeout 10m ``` 3. ### Two target frameworks running at once Since the .NET 9 SDK, a project with `net8.0;net10.0` runs its tests for both frameworks [in parallel by default](https://learn.microsoft.com/dotnet/core/tools/dotnet-test-vstest). On a 2-vCPU runner, the two runs compete for the CPUs and for whatever the suite keeps outside the process, such as the same database or the same port. A test that only fails while the other framework's run touches the same row looks flaky under both. Give each framework its own resources, or run them one after the other. The report needs the `{framework}` token in its file name too, or one framework's report overwrites the other's. ``` dotnet test -p:TestTfmsInParallel=false ``` The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which NUnit test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [NUnit guide](https://mergify.com/learn/flaky-tests/nunit) for the patterns behind flaky NUnit suites on any CI: parallelizableAttribute, TestContext races, and fixture reuse pitfalls. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my NUnit tests pass locally and fail on GitHub Actions? Start with the runner. NUnit sizes its worker pool from the CPU count, so a private repository's 2-vCPU runner runs your parallel fixtures two at a time, and a multi-targeted project runs its frameworks side by side on those same 2 vCPUs. Reproduce with dotnet test -- NUnit.NumberOfTestWorkers=2 and -p:TestTfmsInParallel=false before you call a test flaky. Can I keep the NUnit Console Runner instead of dotnet test? Yes, with one more file. The console runner only writes nunit3 or nunit2 XML, so pass --result=junit.xml;transform=nunit3-junit.xslt with the XSLT NUnit publishes in its nunit-transforms repository, and quote the whole option, because the semicolon is a command separator in most shells. dotnet test with the JUnit logger needs no transform. Why is there no junit.xml after dotnet test? Look in the test project's folder first: LogFilePath is relative to the .csproj, not to the repository root. If the file is nowhere, check whether the project sets TestingPlatformDotnetTestSupport. With it, dotnet test runs the tests on Microsoft.Testing.Platform and ignores --logger without an error. ## Catch the flaky NUnit tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/nunit) --- Source: https://mergify.com/learn/flaky-tests/github-actions/pest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/Pest # Flaky Pest tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, Pest reaches Test Engine through the JUnit report it inherits from PHPUnit: `--log-junit junit.xml` writes it, and the `mergifyio/gha-mergify-ci` action uploads it and sets the job's result with quarantine taken into account. The test step runs Pest with `--ci`, like the GitHub Actions example in Pest's own docs. Below: the workflow, why the two Pest features that sound like retries keep flaky tests out of sight, and the runner details that make Pest suites flake on GitHub Actions. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 5, 2026 Setup ## Run Pest with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `.github/workflows/ci.yml`: Run Pest with --ci and --log-junit, then upload junit.xml with the Mergify action, which sets the job's result. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest env: HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }} steps: - uses: actions/checkout@v7 - uses: shivammathur/setup-php@v2 with: php-version: "8.5" coverage: none - run: composer install --no-interaction --no-progress - name: Run tests id: tests continue-on-error: true run: ./vendor/bin/pest --ci --log-junit junit.xml - name: Mergify CI Upload if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true' uses: mergifyio/gha-mergify-ci@v28 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: junit.xml test_step_outcome: ${{ steps.tests.outcome }} - name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true' run: exit 1 ``` - `--ci` makes Pest [ignore ->only() and run the entire suite](https://github.com/pestphp/docs/blob/5.x/cli-api-reference.md). Without it, an `->only()` committed by mistake shrinks the run to the focused tests, and the report Test Engine receives shrinks with it. - `continue-on-error: true` lets the upload step set the job's result, leaving out the failures of quarantined tests, and `test_step_outcome` tells it how the Pest step really ended. Fork and Dependabot pull requests get no `MERGIFY_TOKEN`: `HAS_MERGIFY_TOKEN` skips the upload there, and the last step keeps a failing run red when nothing was uploaded. - Keep `--tia` off this job. Pest's [test impact analysis](https://github.com/pestphp/docs/blob/5.x/tia.md) replays the stored result of tests whose dependencies did not change instead of running them, and its docs say not to add it to the command that runs your suite on CI. A replayed result is not a new execution, so it tells Test Engine nothing about flakiness. - Pest 5 [requires PHP 8.4 or later](https://github.com/pestphp/docs/blob/5.x/installation.md). Testing several PHP versions in a matrix? Every leg reports the same `GITHUB_JOB`, so add `job_name: test (${{ matrix.php }})` to the upload step, or a test that only fails on one version looks flaky instead of broken. Full reference: [Pest on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/pest). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. [Re-run failed jobs](https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs) starts the job over: setup-php, `composer install`, the Playwright install if you run browser tests, then the whole suite. The attempt that passes is the one the pull request shows. Pest has two options that sound like a retry, and only one of them retries. `--retry` [reorders the suite](https://github.com/pestphp/docs/blob/5.x/filtering-tests.md) so the tests that failed last time run first, and every test still runs once. `->flaky()` reruns a failing test, up to 3 times by default, with `tearDown` and `setUp` in between. Its [retry loop](https://github.com/pestphp/pest/blob/5.x/src/Concerns/Testable.php) catches each failed attempt inside the test and only rethrows when the last one fails, so a test that passes on its second try is a plain pass in the JUnit report. Test Engine never sees it fail. Without `->flaky()`, every failure reaches the report, and every attempt of the job uploads its own, on pull requests and on the default branch. A test that fails on one attempt and passes on another of the same commit is flagged as flaky. Once it is quarantined, the upload step leaves its failure out of the job's result, and the test keeps running and reporting until it is healthy again. Runners ## Why Pest tests flake on GitHub Actions 1. ### One process per core, and 2 cores on a private runner With `--parallel`, Pest [creates a process for each available CPU core](https://github.com/pestphp/docs/blob/5.x/optimizing-tests.md). A [private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) has 2, a public one 4, your laptop maybe 10 or more. Tests that share a static property or one database interleave differently at each width, so an ordering bug that never shows on your machine shows on CI. Reproduce with the runner's count. ``` ./vendor/bin/pest --parallel --processes=2 ``` 2. ### Browser tests waiting 5 seconds on 2 vCPUs Pest's browser plugin [waits 5 seconds](https://github.com/pestphp/docs/blob/5.x/browser-testing.md) before timing out, and the [CI example](https://github.com/pestphp/docs/blob/5.x/continuous-integration.md) runs browser tests with `--parallel`. On a 2-vCPU private runner the browsers, your app and PHP share two cores, and the slowest page of the day misses the wait. Raise the timeout in `tests/Pest.php`, and run with `--trace` so the trace of a failed test lands in `tests/Browser/Traces`, which you can upload as an artifact instead of losing it with the runner. ``` // tests/Pest.php pest()->browser()->timeout(10000); ``` 3. ### Shards that move with every new file `--shard=1/5` across matrix legs balances test files using the timings in `tests/.pest/shards.json`. Files that are not in it yet are [spread evenly across shards](https://github.com/pestphp/docs/blob/5.x/continuous-integration.md), so a pull request that adds tests can move others to another shard, and an ordering problem shows up on a pull request that never touched the failing test. Keep the shard number out of `job_name`: shards run different tests, so they can share one name, and a name per shard would split a test's history every time it moves. 4. ### Xdebug left on from the docs example Pest's own GitHub Actions example sets `coverage: xdebug` on setup-php. [setup-php's README](https://github.com/shivammathur/setup-php) recommends disabling Xdebug when you do not use it, for PHP performance. On a 2-vCPU runner the slower run is what pushes a browser wait or a time-based assertion over its limit. Use `coverage: none` in the job that uploads to Test Engine, and collect coverage in a job of its own. The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which Pest test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [Pest guide](https://mergify.com/learn/flaky-tests/pest) for the patterns behind flaky Pest suites on any CI: higher-order tests, parallel execution, and the Laravel testing stack. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why are my Pest tests flaky only on GitHub Actions? Look at the runner first. --parallel starts one process per core, which is 2 on a private repository, and browser tests share those 2 vCPUs with a 5-second default wait. If you copied Pest's example workflow, Xdebug is on as well, and with --shard a new file can move your test to another shard. Run ./vendor/bin/pest --parallel --processes=2 locally to reproduce. Does pest --retry rerun failed tests? No. It runs the tests that failed last time first, then the rest, and every test runs once. The option that retries is ->flaky(), and its failed attempts never reach the JUnit report, so a test it saves is never flagged. How does a Pest test show up in Test Engine? Under the file's class and the test's description: test('flaky draw') in tests/Unit/FlakyTest.php is reported as Tests.Unit.FlakyTest.flaky draw. The description is part of the name, so rewording it makes a new test as far as Test Engine is concerned. ## Catch the flaky Pest tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/pest) --- Source: https://mergify.com/learn/flaky-tests/github-actions/phpunit [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/PHPUnit # Flaky PHPUnit tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, PHPUnit reaches Test Engine as a JUnit report: `--log-junit junit.xml` writes it, and the `mergifyio/gha-mergify-ci` action uploads it and sets the job's result with quarantine taken into account. PHPUnit writes the report itself, so `phpunit.xml` stays as it is. Below: the workflow, why PHPUnit's own retry hides exactly what Test Engine needs to see, and the runner details that make PHPUnit suites flake on GitHub Actions. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 5, 2026 Setup ## Run PHPUnit with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `.github/workflows/ci.yml`: Write junit.xml with PHPUnit's --log-junit, then upload it with the Mergify action, which sets the job's result. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest env: HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }} steps: - uses: actions/checkout@v7 - uses: shivammathur/setup-php@v2 with: php-version: "8.5" coverage: none - run: composer install --no-interaction --no-progress - name: Run tests id: tests continue-on-error: true run: ./vendor/bin/phpunit --log-junit junit.xml - name: Mergify CI Upload if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true' uses: mergifyio/gha-mergify-ci@v28 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: junit.xml test_step_outcome: ${{ steps.tests.outcome }} - name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true' run: exit 1 ``` - `continue-on-error: true` on the PHPUnit step hands the job's result to the upload step, which leaves out the failures of quarantined tests. `test_step_outcome` passes PHPUnit's own outcome along, so a run that crashed but left a clean-looking report still fails the job. - Pull requests from forks and from Dependabot get no `MERGIFY_TOKEN`. `HAS_MERGIFY_TOKEN` skips the upload there, and the last step fails the job when PHPUnit failed and nothing was uploaded. Without that step, `continue-on-error` would turn those red runs green. - [setup-php's README](https://github.com/shivammathur/setup-php) notes that Xdebug is enabled by default on Ubuntu GitHub Actions images and recommends turning it off for PHP performance. `coverage: none` disables both Xdebug and PCOV. If you need coverage, collect it in a job of its own. - PHPUnit 13 [requires PHP 8.4.1 or later](https://github.com/sebastianbergmann/phpunit/blob/13.4/composer.json), so `php-version` has to match the PHPUnit your lock file installs. Testing several PHP versions in a matrix? Every leg reports the same `GITHUB_JOB`, so add `job_name: test (${{ matrix.php }})` to the upload step. Without it, a test that only fails on one PHP version looks flaky instead of broken. - Running [paratest](https://github.com/paratestphp/paratest)? It takes the same `--log-junit` option. Swap the command in the test step and keep its `id`, and the upload step stays as it is. Full reference: [PHPUnit on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/phpunit). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. [Re-run failed jobs](https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs) starts the job over from the checkout, `composer install` included, and runs every test again to find out whether one of them was unlucky. If the second attempt passes, the pull request goes green, and the failure stays in the log of an attempt nobody opens. PHPUnit 13.3 added a retry of its own, `--retry N` and the `#[Retry]` attribute ([changelog](https://github.com/sebastianbergmann/phpunit/blob/13.3.6/ChangeLog-13.3.md)). It reruns a failing test inside the run and stops at the first success. The console summary still mentions the failed attempt, but the [JUnit logger](https://github.com/sebastianbergmann/phpunit/blob/13.4/src/Logging/JUnit/JunitXmlLogger.php) writes a single testcase from the final attempt, and the failed attempts only add to its time. A test saved by a retry reaches Test Engine as a pass. Leave `--retry` off the job that uploads. With the upload in place, every attempt sends its report, on pull requests and on the default branch, and each one adds to the test's history. A test that fails on one attempt and passes on another of the same commit is flagged as flaky. Once it is quarantined, the upload step leaves its failure out when it sets the job's result, and the test keeps running and reporting until it is healthy again. Runners ## Why PHPUnit tests flake on GitHub Actions 1. ### paratest at the runner's width [paratest](https://github.com/paratestphp/paratest/blob/7.x/src/Options.php) starts one process per CPU core unless you pass `--processes`. A [GitHub-hosted](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) `ubuntu-latest` runner has 2 vCPUs and 8 GB on a private repository and 4 vCPUs and 16 GB on a public one, so the same suite runs on two workers in the company repository and four in the open source one, with different tests sharing each worker's static state. Workers that share one database collide on top of that. paratest sets a `TEST_TOKEN` that is different for every running process: build the test database name from it. ``` # Reproduce a failure from a private-repository runner vendor/bin/paratest --processes=2 ``` 2. ### A new random order on every attempt With `--order-by random` (or the same setting in `phpunit.xml`), each attempt of the job draws a new order, so a test that depends on what ran before it fails on one attempt and passes on the rerun. PHPUnit [prints the seed](https://github.com/sebastianbergmann/phpunit/blob/13.4/src/TextUI/Application.php) as `Random Seed:` at the top of a randomized run. Copy it from the failed attempt's log and replay that order with [\--random-order-seed](https://github.com/sebastianbergmann/phpunit/blob/13.4/src/TextUI/Help.php) before calling the test flaky. ``` ./vendor/bin/phpunit --order-by random --random-order-seed 1762367891 ``` 3. ### A database container that is started but not ready A `services:` database starts before your steps run, but started is not ready. Without a health check, the migrations or the first test that opens a connection can reach a PostgreSQL that is still initializing, and the failure lands on whichever test PHPUnit runs first. [GitHub's own example](https://docs.github.com/en/actions/tutorials/use-containerized-services/create-postgresql-service-containers) adds the health check that makes the job wait for the server. ``` services: postgres: image: postgres env: POSTGRES_PASSWORD: postgres options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 ports: - 5432:5432 ``` 4. ### A library tested without its lock file Without a committed `composer.lock`, `composer install` resolves the dependencies again on every run. A release that lands between two attempts on the same commit makes the second attempt run different code than the first, and the difference reads as a flaky test. [Composer's docs](https://github.com/composer/composer/blob/main/doc/02-libraries.md) recommend committing the lock file for libraries too, and testing against fresh releases in a separate `composer update` job. Give that job its own name, so its failures do not land in the history of the job that runs on the lock file. The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which PHPUnit test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [PHPUnit guide](https://mergify.com/learn/flaky-tests/phpunit) for the patterns behind flaky PHPUnit suites on any CI: test isolation, database transactions, and dataProvider non-determinism. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my PHPUnit tests pass locally but fail on GitHub Actions? On GitHub Actions, look first at the runner's 2 vCPUs on a private repository, which changes how paratest spreads tests across workers, and at random order, which draws a new seed on every attempt. A database container that was not ready and dependencies resolved again without a lock file come next. Replay the failed attempt's seed and run paratest --processes=2 locally before you call the test flaky. Does PHPUnit's --retry work with Test Engine? It runs, but it hides what Test Engine needs. PHPUnit's JUnit report keeps only the final attempt of a retried test, so a test that failed and then passed on retry is uploaded as a pass and never flagged. Drop --retry and #\[Retry\] from the job that uploads, and let quarantine keep the known flaky tests from blocking merges. Why does the PHPUnit step have continue-on-error? So quarantine can work. The upload step decides whether the job passes, and it ignores the failures of quarantined tests. If PHPUnit's failing exit code failed the job first, a quarantined test would still turn the pull request red. Can I upload a report from paratest? Yes. paratest accepts PHPUnit's --log-junit option, so replace the command in the test step, keep the step's id, and leave the upload step unchanged. ## Catch the flaky PHPUnit tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/phpunit) --- Source: https://mergify.com/learn/flaky-tests/github-actions/playwright [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/Playwright # Flaky Playwright tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, Playwright takes two code changes and one secret: wrap the config with `withMergify`, import `test` and `expect` from `@mergifyio/playwright`, and give the test step a `MERGIFY_TOKEN`. The reporter uploads every result and the imported `test` applies quarantine, so a quarantined test that times out does not turn the job red. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated October 28, 2026 Setup ## Run Playwright with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `playwright.config.js`: Wrap the Playwright config with withMergify, which adds the Mergify reporter next to yours. ``` import { defineConfig, devices } from "@playwright/test"; import { withMergify } from "@mergifyio/playwright"; export default withMergify( defineConfig({ testDir: "./tests", reporter: process.env.CI ? "github" : "list", projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }], }), ); ``` `tests/login.spec.ts`: Import test and expect from @mergifyio/playwright in every test and fixture file, which is what applies quarantine. ``` import { expect, test } from "@mergifyio/playwright"; test("logs in", async ({ page }) => { await page.goto("/login"); await expect(page.getByRole("heading", { name: "Sign in" })).toBeVisible(); }); ``` `.github/workflows/ci.yml`: Pass the MERGIFY\_TOKEN secret to the step that runs Playwright. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 24 cache: npm - run: npm ci - run: npx playwright install --with-deps chromium - name: Run tests env: MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }} run: npx playwright test ``` - Both changes are required. With the config wrapped but `test` still imported from `@playwright/test`, the quarantine list is fetched and never applied, and quarantined tests keep failing the job. - Do not pass `--reporter` on the command line: it replaces the configured reporters, the Mergify one included. - Running several browser projects? Set `MERGIFY_PLAYWRIGHT_INCLUDE_PROJECT_IN_TEST_NAME: "true"` on the step, so a test that fails only on WebKit is tracked as broken on WebKit instead of flaky everywhere. - `@playwright/test` 1.56 or later and Node.js 22 or later in the job. - No `continue-on-error`: quarantine changes the expected status of the test inside the run, so Playwright's exit code is already the right one. Full reference: [Playwright on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/playwright). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs is expensive for Playwright: every attempt reinstalls the browsers and their system packages before a single test runs, then replays the whole suite. The config `npm init playwright` generates sets `retries: process.env.CI ? 2 : 0`. Playwright then labels a test that failed and passed on retry as flaky in its HTML report, which is an artifact of that run, and exits 0. The next run starts with no memory of it, and nobody downloads the artifact of a green job. Test Engine keeps that verdict across runs. Every result reaches it, retries included, so a test Playwright quietly retried to green shows up as flaky in Test Engine, and once it is quarantined, its failures stop blocking merges while it keeps running. Runners ## Why Playwright tests flake on GitHub Actions 1. ### One worker on CI, many on your laptop The generated config also sets `workers: process.env.CI ? 1 : undefined`. Locally your tests run in parallel against the same app; on GitHub Actions they run one after the other. A test that relies on data another test created, or on being alone on the server, behaves differently in each. Raising workers on a [2-vCPU private runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) does not help either: browsers starve each other and auto-wait timeouts start to fire. 2. ### Screenshots taken on another system `toHaveScreenshot()` baselines generated on macOS do not match pixels rendered on Ubuntu, and small antialiasing differences can pass on one attempt and fail on the next. Generate and compare baselines in the same Linux image Playwright publishes, on CI and locally. ``` jobs: e2e: runs-on: ubuntu-latest # the tag matches your @playwright/test version container: mcr.microsoft.com/playwright:v1.64.1-noble ``` 3. ### Shards that change with every pull request `--shard=1/4` across matrix legs splits tests by file. Adding tests moves others to another shard, so ordering problems surface on pull requests that never touched the failing test. Name each leg after its environment, never its shard: a test can change shard, and a name per shard would split its history. The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which Playwright test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [Playwright guide](https://mergify.com/learn/flaky-tests/playwright) for the patterns behind flaky Playwright suites on any CI: auto-waiting versus explicit waits, trace-level debugging, and retry config. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why are my Playwright tests flaky only on GitHub Actions? Most of the time it is the CI branch of the generated config: one worker and two retries on CI, all workers and no retries locally. Add a runner with 2 vCPUs on private repositories and screenshots rendered on Ubuntu instead of macOS. Run locally with --workers=1 and the same container image to reproduce. Should I keep retries in playwright.config.ts? Keep them if a red job costs you more than a slow one, but do not treat a retried pass as a pass. Test Engine records the failed attempt, so the test is flagged as flaky even when the job is green. Does sharding break flaky detection? No, as long as the shard index stays out of the job name. Shards run different tests, so they can share one name; a name per shard would split a test's history every time it moves. ## Catch the flaky Playwright tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/playwright) --- Source: https://mergify.com/learn/flaky-tests/github-actions/pytest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/pytest # Flaky pytest tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, pytest needs one package and one secret: install `pytest-mergify` and give the step that runs pytest a `MERGIFY_TOKEN`. The plugin uploads every test result when the session ends, applies quarantine inside the session, and reruns new tests before the pull request can merge. Below: the workflow, how that differs from rerunning the job, and the runner details that make pytest suites flake on GitHub Actions. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated October 28, 2026 Setup ## Run pytest with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `requirements.txt`: Add pytest-mergify next to pytest in the requirements your tests install. ``` pytest pytest-mergify ``` `.github/workflows/ci.yml`: Pass the MERGIFY\_TOKEN secret to the step that runs pytest. Nothing else changes in the workflow. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: actions/setup-python@v7 with: python-version: "3.13" - run: pip install -r requirements.txt - name: Run tests env: MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }} run: pytest ``` - No upload step and no `continue-on-error`. Quarantine is already reflected in pytest's exit code, so the step's result is the job's result. - Running pytest across a matrix? Every leg reports the same `GITHUB_JOB`, so set `MERGIFY_TEST_JOB_NAME: test (${{ matrix.python-version }})` on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken. - The plugin ships CPython wheels only. If a matrix leg runs PyPy, add it as `pytest-mergify; platform_python_implementation == "CPython"` so that leg skips it. - Tests run through tox? tox does not pass the job's environment through. Add `CI`, `MERGIFY_*`, `GITHUB_*` and `RUNNER_*` to `pass_env`, or the plugin cannot see the token or the run it belongs to. Full reference: [pytest on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/pytest). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. When a pytest test fails on GitHub Actions, the button you get is Re-run failed jobs. It runs the job again from the top: checkout, setup-python, the dependency install, then the whole session, to find out whether one test was unlucky. If the second attempt passes, the pull request turns green, and the failure is only visible to whoever opens the first attempt's log. `pytest-rerunfailures` moves the retry inside the job, which is cheaper, but `--reruns 2` applies to every failing test and reports the last attempt. A test that fails one run in three is green in CI and still broken. pytest-mergify reruns inside the session too, with a narrower rule. On a pull request it reruns the tests the target branch does not have yet, as many times as the Prevention budget allows, and fails the run if any attempt fails, so a new flaky test cannot merge. On pushes to the default branch it reruns the tests Test Engine already knows as unhealthy, to keep their status current, without changing the result. Between reruns it holds back the fixture finalizers pytest would run, so a module-scoped fixture is not torn down mid-rerun. Runners ## Why pytest tests flake on GitHub Actions 1. ### Two cores on private repositories [GitHub-hosted](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) `ubuntu-latest` has 4 vCPUs and 16 GB on public repositories, and 2 vCPUs and 8 GB on private ones. `pytest -n auto` starts one xdist worker per CPU, so the same suite runs on four workers in the open source project and on two in the company repository, with a different split of tests per worker. Timeouts tuned on a ten-core laptop fail first. Reproduce with the worker count of the runner that failed, not your machine's. ``` # Reproduce a failure from a private-repo runner pytest -n 2 -p no:randomly ``` 2. ### Service containers that are started but not ready A `services:` database starts before your steps run, but started is not ready. Without a health check, the first tests can reach a PostgreSQL that is still initializing, and the failures land on whichever tests pytest happens to run first. [GitHub's own example](https://docs.github.com/en/actions/tutorials/use-containerized-services/create-postgresql-service-containers) adds the health check; keep it. ``` services: postgres: image: postgres:17 env: POSTGRES_PASSWORD: postgres ports: ["5432:5432"] options: >- --health-cmd pg_isready --health-interval 5s --health-timeout 5s --health-retries 10 ``` 3. ### A rerun that installs different packages setup-python's `cache: pip` caches downloads, not your environment, and an unpinned requirements file is resolved again on every run. A dependency released between two attempts on the same commit makes the second attempt run different code than the first, and the difference reads as a flaky test. Install from a lock file (`uv.lock`, `poetry.lock`, pip-compile output) so a rerun of the same SHA installs the same packages. 4. ### Runners in UTC Hosted runners run in UTC. A test that builds "today" from `datetime.now()` and compares it with a date computed elsewhere passes all day in Paris or San Francisco and fails on CI in the hour around midnight UTC. Freeze the clock in the test (`time-machine`, `freezegun`) instead of trusting the runner's. The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which pytest test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [pytest guide](https://mergify.com/learn/flaky-tests/pytest) for the patterns behind flaky pytest suites on any CI: fixture teardown races, xdist ordering, and Hypothesis seed non-determinism. ## Other frameworks on GitHub Actions - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my pytest tests pass locally but fail on GitHub Actions? The usual suspects on GitHub Actions: fewer CPUs than your laptop (2 on a private repository), which changes how pytest-xdist spreads the tests; a service container that was not ready yet; dependencies resolved again on every run; and a runner clock in UTC. Reproduce with the runner's worker count and the same lock file before you call it flaky. Do I need to upload a JUnit XML file from pytest? No. pytest-mergify uploads the results itself when the session ends, so the workflow has no upload step and no continue-on-error. If another tool still reads a JUnit file, keep --junitxml for it. Does pytest-mergify slow down my GitHub Actions job? The upload happens once, at the end of the session. Reruns only happen within the budgets you set: on a pull request for tests that are new, on the default branch for tests already known as unhealthy. That is a few tests, not the session, which is what Re-run failed jobs costs. ## Catch the flaky pytest tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/pytest) --- Source: https://mergify.com/learn/flaky-tests/github-actions/rspec [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/RSpec # Flaky RSpec tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, RSpec needs a gem, one require and one secret: add `rspec-mergify` to the test group, require it from `.rspec`, and give the step that runs RSpec a `MERGIFY_TOKEN`. The gem uploads every example's result, applies quarantine inside the run, and reruns new examples before the pull request can merge. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated October 28, 2026 Setup ## Run RSpec with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `Gemfile`: Add rspec-mergify to the test group of the Gemfile, next to rspec or rspec-rails. ``` group :test do gem "rspec" gem "rspec-mergify" end ``` `.rspec`: Require rspec\_mergify from .rspec, since Bundler does not load it. ``` --require spec_helper --require rspec_mergify ``` `.github/workflows/ci.yml`: Pass the MERGIFY\_TOKEN secret to the step that runs RSpec. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: ruby/setup-ruby@v1 with: ruby-version: "3.4" bundler-cache: true - name: Run tests env: MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }} run: bundle exec rspec ``` - `bundle exec rspec` does not call `Bundler.require`, and the gem's entry file is `rspec_mergify`. Without the `--require` line, RSpec runs as usual and nothing reaches Test Engine. - Lockfile generated on a Mac? Run `bundle lock --add-platform x86_64-linux` and commit it, or `bundler-cache` fails on the Linux runner before any spec runs. - Running RSpec across a matrix? Every leg reports the same `GITHUB_JOB`, so set `MERGIFY_TEST_JOB_NAME: test (${{ matrix.ruby-version }})` on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken. - CRuby 3.1 to 4.0. A JRuby or `ruby-head` leg runs the suite but uploads nothing; give that leg a JUnit report from `rspec_junit_formatter` and the upload action instead. - No `continue-on-error`: a quarantined example that fails is marked pending, so RSpec's exit code is already the right one. Full reference: [RSpec on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/rspec). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs replays the job from the top: Ruby setup, `bundle install` from cache, any service containers and database setup, then every example, to retry the one that failed. `rspec-retry` retries the example in place and reports the attempt that passed. The flake is absorbed and nothing records it. rspec-mergify counts those attempts instead of hiding them. On a pull request it reruns the examples the target branch does not have yet, right after its own run, within the Prevention budget, and a failure among rspec-retry's tries counts too: the example is flaky and RSpec reports it failed. On the default branch it reruns the examples Test Engine knows as unhealthy without changing the result. Runners ## Why RSpec tests flake on GitHub Actions 1. ### A new random order on every attempt With `config.order = :random`, each attempt of the job runs a new seed, so an order-dependent spec fails on one attempt and passes on the rerun. The seed is printed in the failed attempt's log as "Randomized with seed". Reproduce with it before calling the spec flaky. ``` bundle exec rspec --seed 41853 --bisect ``` 2. ### Capybara's two-second wait on a two-core runner `Capybara.default_max_wait_time` is 2 seconds. Headless Chrome on a [2-vCPU private runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners), sharing it with Rails and Postgres, takes longer to render than your laptop, and system specs time out on whichever page is slowest that day. Raise the wait for CI rather than adding sleeps. ``` # spec/support/capybara.rb Capybara.default_max_wait_time = ENV["CI"] ? 5 : 2 ``` 3. ### Database setup racing the service container In a Rails app, `bin/rails db:prepare` against a Postgres [service container](https://docs.github.com/en/actions/tutorials/use-containerized-services/create-postgresql-service-containers) with no health check can run while the server is still starting. Sometimes it fails outright; sometimes it half-succeeds and the first specs fail on missing tables. A health check makes the job wait for the server before any step runs. ``` services: postgres: image: postgres:17 env: POSTGRES_PASSWORD: postgres ports: ["5432:5432"] options: >- --health-cmd pg_isready --health-interval 5s --health-timeout 5s --health-retries 10 ``` The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which RSpec test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [RSpec guide](https://mergify.com/learn/flaky-tests/rspec) for the patterns behind flaky RSpec suites on any CI: order-dependent specs, database\_cleaner strategies, and rspec-retry pitfalls. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my RSpec specs pass locally and fail on GitHub Actions? On GitHub Actions the usual causes are a different random seed on every attempt, Capybara timeouts on a 2-vCPU runner, and a database that was not ready when the suite started. Re-run locally with the seed from the failed attempt's log first. Does rspec-mergify replace rspec-retry? It does not need to. If you keep rspec-retry, rspec-mergify counts its attempts: a new example that only passed on retry is flaky and fails the pull request. Most teams drop rspec-retry once quarantine handles the known flaky specs. Do I need rspec\_junit\_formatter on GitHub Actions? Not with rspec-mergify, which uploads the results itself. Keep the formatter only for legs the gem does not support, such as JRuby, and upload that report with the Mergify upload action. ## Catch the flaky RSpec tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/rspec) --- Source: https://mergify.com/learn/flaky-tests/github-actions/rust [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/Rust # Flaky Rust tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, Rust tests reach Test Engine as a JUnit report: `cargo-nextest` writes it from a `ci` profile, and the `mergifyio/gha-mergify-ci` action uploads it and decides the job's result with quarantine taken into account. Two lines of that profile matter on CI: the report path, and `fail-fast`, which nextest turns on by default and which would stop the run before every test has a result to report. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 5, 2026 Setup ## Run Rust with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `.config/nextest.toml`: Add a ci profile that runs every test and writes a JUnit report to target/nextest/ci/junit.xml. ``` [profile.ci] # Run all tests regardless of failures. fail-fast = false [profile.ci.junit] path = "junit.xml" ``` `.github/workflows/ci.yml`: Install cargo-nextest, run the ci profile, then upload the report with the Mergify action, which sets the job's result. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest env: HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }} steps: - uses: actions/checkout@v7 - name: Install cargo-nextest uses: taiki-e/install-action@nextest - name: Run tests id: tests continue-on-error: true run: cargo nextest run --profile ci - name: Mergify CI Upload if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true' uses: mergifyio/gha-mergify-ci@v28 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: target/nextest/ci/junit.xml test_step_outcome: ${{ steps.tests.outcome }} - name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true' run: exit 1 ``` - `fail-fast = false` is [nextest's own advice for a CI profile](https://nexte.st/docs/configuration/). With the default, the run stops scheduling tests after the first failure, and a test that never ran has no result for Test Engine. - The JUnit `path` is relative to the profile's directory, so the `ci` profile [writes target/nextest/ci/junit.xml](https://nexte.st/docs/machine-readable/junit/). That is the `report_path` the upload step needs. - `continue-on-error: true` on the test step is what lets quarantine work: the upload step decides whether the job passes, ignoring failures of quarantined tests. `test_step_outcome` tells it when the test step itself failed, so a crash that left a clean report does not pass for green. - Pull requests from forks and from Dependabot get no `MERGIFY_TOKEN`. `HAS_MERGIFY_TOKEN` skips the upload there, and the last step fails the job when tests failed and nothing was uploaded, or `continue-on-error` would let those failures through. - The hosted Ubuntu image ships a Rust toolchain but not cargo-nextest. [taiki-e/install-action](https://nexte.st/docs/installation/pre-built-binaries/) downloads nextest's prebuilt binary instead of compiling it with `cargo install`. - Testing several toolchains in a matrix? Every leg reports the same `GITHUB_JOB`, so set the upload step's `job_name` input per leg, such as `job_name: test (${{ matrix.toolchain }})`. Without it, the legs' results merge into one test, and a test that only fails on beta looks flaky instead of broken. Full reference: [Rust on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/rust). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs starts the job again on a new runner. Unless you cache `target/`, the crate and its dependencies compile from scratch before every test binary runs again, all to retry one test. [nextest's own retries](https://nexte.st/docs/features/retries/) are cheaper: set `retries = 2` in the profile, or pass `--retries 2`, and a failing test is run again on its own. A test that passes on a retry is marked FLAKY, and by default, if nothing else failed, the run exits 0. The failed attempts are kept in the JUnit report as elements, inside a test case that reads as passed. Since nextest 0.9.131, `flaky-result = "fail"` fails the run on a flaky test instead. With Test Engine, every upload adds to each test's history on the default branch and on pull requests, which is how a test that passes and fails on the same commit gets flagged. Once a flaky test is quarantined, the upload step ignores its failure when it sets the job's result, and the test keeps running and reporting until it is healthy again. Runners ## Why Rust tests flake on GitHub Actions 1. ### One process per test, two at a time nextest [runs each test in its own process](https://nexte.st/docs/design/why-process-per-test/), and [test-threads](https://nexte.st/docs/configuration/reference/) defaults to the number of CPUs: [2 on a private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners), 4 on a public one. Statics and environment variables are no longer shared between tests the way they are under `cargo test`, but anything outside the process still is, like a fixed port or the database in a service container. How often two tests collide over it depends on how many run at once, so reproduce with the runner's count. ``` cargo nextest run --profile ci --test-threads 2 ``` 2. ### A hung test that leaves no report nextest [marks a test SLOW after 60 seconds](https://nexte.st/docs/features/slow-tests/) but lets it run unless the profile sets `terminate-after`. A test that deadlocks on a two-core runner then runs until the job's [timeout-minutes](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idtimeout-minutes), 360 by default, cancels the job. The upload step only runs on [success() or failure()](https://docs.github.com/en/actions/reference/workflows-and-actions/expressions#status-check-functions), not on a cancelled job, so that run reports nothing. Terminate the test instead, and it fails under its own name in the report. ``` [profile.ci] fail-fast = false slow-timeout = { period = "60s", terminate-after = 3 } ``` 3. ### Shards that change with every pull request [\--partition slice:1/3](https://nexte.st/docs/ci-features/partitioning/) deals tests out round-robin, after filters, so adding a test moves others to a different shard, next to different neighbors. A test that collides with a neighbor over a port or a file can start failing on a pull request that never touched it. `hash:1/3` keeps each test in the same bucket whatever is added. Either way, keep the shard index out of `job_name`: a name per shard would split a test's history. ``` cargo nextest run --profile ci --partition hash:1/3 ``` The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which Rust test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [Rust guide](https://mergify.com/learn/flaky-tests/rust) for the patterns behind flaky Rust suites on any CI: test-thread races, async runtime drift, and cargo test parallelism. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) ## Frequently asked questions Why do my Rust tests pass with cargo test and fail with nextest on GitHub Actions? cargo test runs a binary's tests as threads of one process, and nextest runs each test in its own process, as many at once as the runner has CPUs (2 on a private repository). State inside the process is shared under cargo test and not under nextest, while a port or a database is shared under both. Reproduce with cargo nextest run --test-threads 2. Can I keep cargo test instead of cargo-nextest? Yes. cargo2junit converts cargo test's JSON output to JUnit, which needs RUSTC\_BOOTSTRAP=1 on a stable toolchain. Add set -o pipefail to the step, or it reports cargo2junit's exit status instead of the test run's, and a failing run can upload as green. What name does a Rust test get in Test Engine? nextest writes one test suite per test binary and one test case per test, so a test is named after its binary and its path, such as my-crate.tests::parses\_header. Renaming the crate, the binary or the module starts a new history for it. Why does the upload step need test\_step\_outcome? With continue-on-error on the test step, the upload step decides the job's result. test\_step\_outcome tells it that cargo nextest failed, so a crash that left a partial but clean junit.xml fails the job instead of passing it. ## Catch the flaky Rust tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/rust) --- Source: https://mergify.com/learn/flaky-tests/github-actions/testng [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/TestNG # Flaky TestNG tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, TestNG results reach Test Engine as the JUnit XML report Maven Surefire writes for the run. The pom below carries the Surefire configuration from the Mergify docs, and the `mergifyio/gha-mergify-ci` action uploads the report and decides the job's result with quarantine taken into account. Two details decide what Test Engine sees: which file the upload pattern picks up, and what a retry analyzer did to a failure before it was written down. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated November 5, 2026 Setup ## Run TestNG with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `pom.xml`: Pin Surefire and set TestNG's listeners as the Mergify docs do. Surefire writes the JUnit XML report that the upload step reads. ``` 4.0.0 com.example te-demo-testng 1.0-SNAPSHOT 21 UTF-8 org.testng testng 7.12.0 test org.apache.maven.plugins maven-surefire-plugin 3.5.3 usedefaultlisteners false listener org.testng.reporters.JUnitReportReporter ${project.build.directory}/surefire-reports ``` `.github/workflows/ci.yml`: Run Maven with continue-on-error, then upload Surefire's report with the Mergify action, which sets the job's result. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest env: HAS_MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN != '' }} steps: - uses: actions/checkout@v7 - uses: actions/setup-java@v6 with: distribution: temurin java-version: 21 cache: maven - name: Run tests id: tests continue-on-error: true run: mvn test - name: Mergify CI Upload if: (success() || failure()) && env.HAS_MERGIFY_TOKEN == 'true' uses: mergifyio/gha-mergify-ci@v28 with: token: ${{ secrets.MERGIFY_TOKEN }} report_path: target/surefire-reports/*.xml test_step_outcome: ${{ steps.tests.outcome }} - name: Fail when tests failed and no report was uploaded if: steps.tests.outcome == 'failure' && env.HAS_MERGIFY_TOKEN != 'true' run: exit 1 ``` - The file the upload step sends is Surefire's own `target/surefire-reports/TEST-TestSuite.xml`: every test of the run in one report, each named by class and method, such as `com.example.FlakyTest.sometimesFails`. The `JUnitReportReporter` listener writes per-class copies one level down, in `junitreports/`, which `target/surefire-reports/*.xml` does not reach. Keep the pattern that shallow: those copies describe the same tests, and a recursive pattern would send each one twice. `usedefaultlisteners` set to false turns off the HTML and XML reports [TestNG attaches by default](https://maven.apache.org/surefire/maven-surefire-plugin/examples/testng.html); on our runs, Surefire's report was the only file the pattern matched. - `continue-on-error: true` on the Maven step hands the job's result to the upload step, which leaves out failures of quarantined tests. `test_step_outcome` passes Maven's own result along, so a build that failed for a reason the report does not show, such as a test class that does not compile, still fails the job. - Pull requests from forks and from Dependabot run without `MERGIFY_TOKEN`. `HAS_MERGIFY_TOKEN` skips the upload there, and the last step fails the job when tests failed and nothing was uploaded, since `continue-on-error` would otherwise let those failures through. - The pom pins Surefire 3.5.3, as the Mergify docs do, and that version runs TestNG through its own TestNG provider. Starting with Surefire 3.6.0, TestNG [runs through the TestNG JUnit Platform engine](https://maven.apache.org/surefire/maven-surefire-plugin/examples/testng.html) instead. If your build already uses 3.6 or later, look at the file names in `target/surefire-reports/` after one run and point `report_path` at the `TEST-*.xml` files you find there. - On Gradle, call `useTestNG()` in the `test` task, run `./gradlew test` and upload `build/test-results/test/*.xml`. Gradle writes those JUnit XML reports itself, with no listener to configure. Full reference: [TestNG on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/testng). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs starts the job from the top. The `cache: maven` input of [setup-java](https://github.com/actions/setup-java) brings `~/.m2/repository` back, but Maven compiles the tests again and Surefire runs the whole suite to retry the one method that failed. The pull request keeps whichever attempt comes back green. TestNG retries inside the run when a test has a [retry analyzer](https://testng.org/#_rerunning_failed_tests): after each failure, it asks the `IRetryAnalyzer` bound with `@Test(retryAnalyzer = ...)` whether to run the test again. We bound one that allows two retries to a test that fails 40% of the time and ran it through the setup above. The test failed, passed on its retry, and Maven ended with `BUILD SUCCESS`. Surefire's report listed the test twice: the failed attempt as ``, carrying the assertion message, and the retry as a plain pass. Test Engine stored one skip and one pass for that run, and no failure. Test Engine flags a test as flaky when it fails and passes on the same commit, and a skip is not a failure. So take the retry analyzer off the runs you upload. Every failure then reaches Test Engine, which records each test's result on pull requests and on the default branch. Once a flaky test is quarantined, the upload step leaves its failure out of the job's result, and the test keeps running and reporting until it is healthy again. Runners ## Why TestNG tests flake on GitHub Actions 1. ### Five threads on two vCPUs TestNG's thread count does not follow the machine. Set `parallel` and Surefire runs your tests on [5 threads](https://maven.apache.org/surefire/maven-surefire-plugin/examples/testng.html) unless you change `threadCount`. On a [private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners), those 5 threads share 2 vCPUs, where a public repository's runner has 4 and your laptop likely more. A method that gives another thread a fixed delay to finish its work gets less CPU inside that delay than it does locally, and fails on CI only. Classes whose methods share fields can opt out of method-level parallelism: [`singleThreaded = true`](https://testng.org/#_annotations) keeps all of a class's methods on one thread, even under `parallel="methods"`. ``` @Test(singleThreaded = true) public class CartTest { // every test method of this class runs on the same thread } ``` 2. ### Test JVMs counted from the runner's cores With `forkCount=1C`, Surefire [multiplies the number by the CPU cores](https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html): 2 test JVMs at once on a [private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners), 4 on a public one, one per core on your laptop. With `reuseForks` on by default, test classes are handed to the forks one by one, so which classes share a JVM, and with it every static field, changes with the core count. A class that only passes because another one warmed a cache before it can fail on CI first. The forks also run at the same time, so give each one its own database schema or port with `${surefire.forkNumber}`, and reproduce with the runner's count, `mvn test -DforkCount=2`. ``` 1C test_${surefire.forkNumber} ``` 3. ### A hung test that holds the job for six hours A test that never returns keeps the job running until GitHub cancels it, [after 360 minutes](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idtimeout-minutes) by default. TestNG's [`timeOut`](https://testng.org/#_parallelism_and_time_outs) fails a method that runs past its limit, in parallel and sequential runs alike, but only on the methods that carry one. Surefire's [`forkedProcessTimeoutInSeconds`](https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html), `surefire.timeout` on the command line, covers the whole run: Surefire kills the test JVM after that many seconds, and the job fails in minutes instead of hours. Set it well above your slowest green run, or a busy runner trips it. ``` - name: Run tests id: tests continue-on-error: true run: mvn test -Dsurefire.timeout=1200 ``` The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which TestNG test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [TestNG guide](https://mergify.com/learn/flaky-tests/testng) for the patterns behind flaky TestNG suites on any CI: dependency groups, parallel methods, and retry analyzers done wrong. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Vitest](https://mergify.com/learn/flaky-tests/github-actions/vitest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why do my TestNG tests pass locally and fail on GitHub Actions? Start with the runner. A private repository's runner has 2 vCPUs, so a parallel suite's 5 threads share 2 CPUs there, and with forkCount=1C, Surefire starts 2 test JVMs instead of one per core of your laptop. Reproduce with mvn test -DforkCount=2, then run the failing class on its own and with the rest of the suite, to see whether it depends on state another class leaves behind. Can I keep my IRetryAnalyzer? Not on the runs you upload. The attempt it retries is written to Surefire's report as skipped and the retry as a pass, so Test Engine records a skip and a pass and never sees the failure it needs to flag the test. If the analyzer has to stay in the code, make retry() return false when the CI environment variable is set: GitHub Actions sets it to true on every run. Why do tests that depend on a flaky test show up as skipped? With dependsOnMethods or dependsOnGroups, a method whose dependency failed is not run and is marked SKIP. On an attempt where the flaky test fails, its dependents reach Test Engine as skipped, not failed, and quarantining the flaky test does not change that. If they only need to run after it, alwaysRun = true makes the dependency soft: they run even when it failed. ## Catch the flaky TestNG tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/testng) --- Source: https://mergify.com/learn/flaky-tests/github-actions/vitest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/[GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions)/Vitest # Flaky Vitest tests on GitHub Actions. Set up, retried right, quarantined. On GitHub Actions, Vitest needs one reporter and one secret: add `@mergifyio/vitest` to your reporters and give the test step a `MERGIFY_TOKEN`. The reporter uploads every result, applies quarantine in every project of the run, and reruns new tests before the pull request can merge. The one GitHub Actions detail to get right is the reporter list, because setting it drops the annotations Vitest writes on pull requests. By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated October 28, 2026 Setup ## Run Vitest with Test Engine on GitHub Actions Create a Mergify application key with the `ci` scope and store it as the `MERGIFY_TOKEN` secret. Then: `vitest.config.js`: Add the Mergify reporter and keep the default and github-actions reporters, which an explicit list replaces. ``` import { defineConfig } from "vitest/config"; export default defineConfig({ test: { reporters: [ "default", ...(process.env.GITHUB_ACTIONS === "true" ? ["github-actions"] : []), "@mergifyio/vitest", ], }, }); ``` `.github/workflows/ci.yml`: Pass the MERGIFY\_TOKEN secret to the step that runs Vitest. ``` name: CI on: push: branches: [main] pull_request: jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 24 cache: npm - run: npm ci - name: Run tests env: MERGIFY_TOKEN: ${{ secrets.MERGIFY_TOKEN }} run: npm test # vitest run ``` - Setting `reporters` replaces Vitest's defaults. Keep `"default"` for the console and `"github-actions"` on GitHub Actions: that reporter is what turns a failure into an annotation on the pull request's diff. - The reporter needs Node.js 22 or later in the job. With an older Node.js it reports nothing and prints no Mergify line. - Running Vitest across a matrix? Every leg reports the same `GITHUB_JOB`, so set `MERGIFY_TEST_JOB_NAME: test (${{ matrix.node-version }})` on each leg. Without it, the legs' results merge into one test, and a test that only fails on one version looks flaky instead of broken. - No `continue-on-error` and no upload step: quarantine is applied inside the run, so Vitest's exit code is already the right one. Full reference: [Vitest on docs.mergify.com](https://docs.mergify.com/test-engine/test-frameworks/vitest). Retries ## Re-run failed jobs versus retrying the test GitHub Actions retries at the job level. Re-run failed jobs, or `gh run rerun --failed`, starts the failed jobs again with the same `GITHUB_SHA` and `GITHUB_REF`, up to 30 days after the run and up to 50 times per run. Each attempt repeats the whole job: checkout, toolchain setup, dependency install, then every test, including the ones that passed. Re-run failed jobs repeats `npm ci` and the whole Vitest run for one test that may have been unlucky, and the attempt that passes is the one the pull request shows. Vitest's own `retry` option is cheaper: it retries the failing test inside the run and reports it as passed when an attempt succeeds. The test is still flaky, and nothing outside that log says so. `@mergifyio/vitest` reruns with a rule instead of a blanket count. On a pull request, once the run is over, it starts a Vitest run of its own on the files holding tests the target branch does not have yet, repeats those tests within the Prevention budget, and fails the run if any attempt fails. On the default branch it reruns the tests Test Engine knows as unhealthy without changing the result. The reporter's Mergify CI section in the log names what it caught. Runners ## Why Vitest tests flake on GitHub Actions 1. ### Fewer workers than on your machine Vitest sizes its pool from the CPUs it sees. A [private-repository runner](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) has 2, a public one 4, your laptop maybe 12. Fewer workers means each one runs more files back to back, which exposes state a file leaves behind for the next one in the same worker, and lengthens each file's wall time against the default 5-second `testTimeout`. Reproduce with the runner's worker count. ``` npx vitest run --maxWorkers=2 ``` 2. ### Shards that change with every pull request `--shard=1/3` across matrix legs splits the suite by file. Adding a file moves others to a different shard, so a test that depends on what ran before it in the same worker can start failing on a pull request that never touched it. Keep the shard index out of `MERGIFY_TEST_JOB_NAME`: a test moves between shards, and a name per shard would split its history. 3. ### A Node.js version that moves under you `node-version: 24` takes whichever 24.x the runner image has cached, and GitHub updates the images every week. Two attempts on the same commit can run on different images and different Node.js patch releases. When a failure only shows on one of them, pin the exact version (`node-version-file: .nvmrc`) so a rerun runs the same runtime. 4. ### Runners in UTC Hosted runners run in UTC. A test that formats `new Date()` as a calendar day fails on CI in the hour around midnight UTC and nowhere else. Set the clock with `vi.setSystemTime()` in the test. The missing layer ## GitHub Actions doesn't know what a test is. GitHub Actions tells you a job failed. Test Engine tells you which Vitest test failed, whether it failed on the same commit before, and whether it should block the merge. Read [what Test Engine adds to GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions), or the [Vitest guide](https://mergify.com/learn/flaky-tests/vitest) for the patterns behind flaky Vitest suites on any CI: thread vs fork pool, mock hoisting, and shared-state snapshot flakiness. ## Other frameworks on GitHub Actions - [pytest](https://mergify.com/learn/flaky-tests/github-actions/pytest) - [Playwright](https://mergify.com/learn/flaky-tests/github-actions/playwright) - [RSpec](https://mergify.com/learn/flaky-tests/github-actions/rspec) - [Jest](https://mergify.com/learn/flaky-tests/github-actions/jest) - [Go](https://mergify.com/learn/flaky-tests/github-actions/go) - [Cypress](https://mergify.com/learn/flaky-tests/github-actions/cypress) - [JUnit](https://mergify.com/learn/flaky-tests/github-actions/junit) - [TestNG](https://mergify.com/learn/flaky-tests/github-actions/testng) - [Minitest](https://mergify.com/learn/flaky-tests/github-actions/minitest) - [MSTest](https://mergify.com/learn/flaky-tests/github-actions/mstest) - [NUnit](https://mergify.com/learn/flaky-tests/github-actions/nunit) - [Pest](https://mergify.com/learn/flaky-tests/github-actions/pest) - [PHPUnit](https://mergify.com/learn/flaky-tests/github-actions/phpunit) - [Rust](https://mergify.com/learn/flaky-tests/github-actions/rust) ## Frequently asked questions Why did my Vitest failures stop showing on the pull request diff? Annotations come from the github-actions reporter, which Vitest only adds by default when you have not set reporters yourself. Once you list reporters to add the Mergify one, add "github-actions" back for GitHub Actions runs, as in the configuration above. Does @mergifyio/vitest work with Vitest projects and browser mode? Quarantine applies to every project of the run, browser mode and custom pools included. If two projects run the same test files, run them in separate jobs or steps, each with its own MERGIFY\_TEST\_JOB\_NAME, so each test keeps one history. Do I need vitest run in CI, or is vitest enough? GitHub Actions sets CI=true, and Vitest runs once instead of watching when it sees it. vitest run says the same thing explicitly, which is clearer to whoever reads the workflow. ## Catch the flaky Vitest tests in your GitHub Actions runs. Mergify merges more than half a million pull requests a month without breaking main. [Start a free 14-day trial](https://dashboard.mergify.com/)[Read the setup docs](https://docs.mergify.com/test-engine/test-frameworks/vitest) --- Source: https://mergify.com/learn/flaky-tests/go [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/Go # Flaky tests in Go. Named, fixed, and quarantined. Flaky Go suites are not random. They follow patterns: t.Parallel races on package state, leaked goroutines, map iteration assumptions, real-clock timing, leaked httptest servers. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Published April 2026 Example PR comment from the Mergify bot detecting a flaky Go test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky Go suite: t.Parallel() racing on shared package state, goroutine leaks across test boundaries, map iteration order assumptions, time-based assertions without testing/synctest, httptest.Server connection leaks, TestMain teardown bypassed by os.Exit, subtest plus t.Parallel interleaving surprises, and CI retries hiding flaky Go test failures. Each one has a minimal reproducer, a fix in stock Go, and a Mergify Test Engine detection strategy below. ## Why are Go tests flaky? Go's testing model is intentionally minimal: a flat `_test.go` file in each package, a single `go test ./...` command, parallelism opt-in via [`t.Parallel()`](https://pkg.go.dev/testing#T.Parallel). The minimalism is the appeal. It is also why every flake category boils down to "shared mutable state that the test author did not see as shared." Three Go-specific facets do most of the damage. Package-level `var` declarations are global by language design and survive across every test in the package. Goroutines started by a test outlive the test unless the author explicitly waits for them. Maps iterate in [unspecified order](https://go.dev/ref/spec#For_statements) and the runtime exploits that to keep developers honest, which means an assertion on the first key of a map is a flake the moment the map grows. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of Go suites: t.Parallel() racing on shared package state, goroutine leaks across test boundaries, map iteration order assumptions, time-based assertions without testing/synctest, httptest.Server connection leaks, TestMain teardown bypassed by os.Exit, subtest plus t.Parallel interleaving surprises, and CI retries hiding flaky Go test failures. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01t.Parallel() racing on shared package state](#pattern-t-parallel-racing-on-shared-package-state) 2. [02Goroutine leaks across test boundaries](#pattern-goroutine-leaks-across-test-boundaries) 3. [03Map iteration order assumptions](#pattern-map-iteration-order-assumptions) 4. [04Time-based assertions without testing/synctest](#pattern-time-based-assertions-without-testing-synctest) 5. [05httptest.Server connection leaks](#pattern-httptest-server-connection-leaks) 6. [06TestMain teardown bypassed by os.Exit](#pattern-testmain-teardown-bypassed-by-os-exit) 7. [07Subtest plus t.Parallel interleaving surprises](#pattern-subtest-plus-t-parallel-interleaving-surprises) 8. [08CI retries hiding flaky Go test failures](#pattern-ci-retries-hiding-flaky-go-test-failures) Pattern 1 ### t.Parallel() racing on shared package state Symptom. A test that has been green for months starts failing intermittently after another test in the same package adopts t.Parallel(), and the failure mentions a value the failing test never set. Root cause. Calling [`t.Parallel()`](https://pkg.go.dev/testing#T.Parallel) tells `go test` to run that test alongside the next parallel test in the same package. Anything they share (a package-level `var`, a singleton client, a shared temp file path, an environment variable read at init) becomes a race. Tests that worked sequentially fail under parallelism without any code change. ``` var lastUserID string // package-level state func TestCreateUser(t *testing.T) { t.Parallel() lastUserID = createUser(t).ID if lastUserID == "" { t.Fatal("expected a user ID") } } func TestNotifyLastUser(t *testing.T) { t.Parallel() // races with TestCreateUser; sometimes "" sometimes a real ID notify(lastUserID) } ``` Fix. Treat package-level mutable state as a smell in tested packages. Pass values through `*testing.T` helpers or local variables so each parallel test owns what it touches. For unavoidable shared external state (a database, a temp directory), give each parallel test its own slice via `t.TempDir()`, a row scoped by `t.Name()`, or a transactional fixture. ``` func TestCreateUser(t *testing.T) { t.Parallel() id := createUser(t).ID if id == "" { t.Fatal("expected a user ID") } } func TestNotifyUser(t *testing.T) { t.Parallel() id := createUser(t).ID notify(id) } ``` With Mergify. Test Engine catches the parallel-only signature: a test only fails when run with -parallel > 1 and passes under -parallel=1 reruns. The dashboard tags it as parallelism-sensitive so the shared-state root cause is the obvious lead. Pattern 2 ### Goroutine leaks across test boundaries Symptom. A test passes, the next test fails with \`connection refused\` or a panic from a goroutine that the failing test never started. Root cause. Tests start goroutines (a worker, a server, a poller) and finish before those goroutines do. The runtime keeps them running into the next test, where they continue to push to a channel, write to a file, or hit a network endpoint that the new test owns. The new test sees behavior it cannot explain. ``` func TestPollMetrics(t *testing.T) { go func() { for { time.Sleep(10 * time.Millisecond) metrics.Push() // still running after t returns } }() // no cleanup } func TestMetricsAreEmpty(t *testing.T) { if got := metrics.Len(); got != 0 { t.Fatalf("expected empty, got %d", got) // sometimes fails } } ``` Fix. Use a `context.WithCancel` tied to [`t.Cleanup`](https://pkg.go.dev/testing#T.Cleanup) so every goroutine the test starts is cancelled when the test ends. Add [go.uber.org/goleak](https://pkg.go.dev/go.uber.org/goleak) in `TestMain` to fail loudly when a leak ships. ``` func TestPollMetrics(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) t.Cleanup(cancel) go func() { ticker := time.NewTicker(10 * time.Millisecond) defer ticker.Stop() for { select { case <-ctx.Done(): return case <-ticker.C: metrics.Push() } } }() } ``` With Mergify. Test Engine surfaces the cross-test signature: test B fails consistently when run after test A and never alone. The dashboard groups failures by their predecessor so the leaking-goroutine pattern is easy to find. Pattern 3 ### Map iteration order assumptions Symptom. A test passes hundreds of times locally and fails once a week on CI with an assertion against the first element of a slice built from a map. Root cause. Go intentionally [randomizes map iteration order](https://go.dev/blog/maps#iteration-order) to discourage code that depends on it. A test that iterates a map and asserts on the first value implicitly assumes an order Go does not promise. The randomization is per-process, so the test can pass on every run for months until the seed lands on the unhappy ordering. ``` func TestNamesContainsAlice(t *testing.T) { users := map[string]int{"alice": 1, "bob": 2, "carol": 3} first := "" for name := range users { first = name break } if first != "alice" { t.Fatalf("expected alice, got %q", first) // fails ~66% of the time } } ``` Fix. Sort before asserting. Build a slice from the map's keys (or values), `sort.Strings` it, then compare. For "the map contains X" tests, prefer an explicit lookup over iterating. ``` func TestNamesContainsAlice(t *testing.T) { users := map[string]int{"alice": 1, "bob": 2, "carol": 3} if _, ok := users["alice"]; !ok { t.Fatal("expected alice in users") } } // or, when order matters, sort: func TestNamesSorted(t *testing.T) { users := map[string]int{"alice": 1, "bob": 2, "carol": 3} names := slices.Sorted(maps.Keys(users)) if !slices.Equal(names, []string{"alice", "bob", "carol"}) { t.Fatalf("got %v", names) } } ``` With Mergify. Test Engine tracks pass rates per test on the default branch. A test that fails once every few hundred runs with a string-comparison error against a known map iteration is the textbook signature; the dashboard surfaces it under map-iteration confidence drops. Pattern 4 ### Time-based assertions without testing/synctest Symptom. A test that asserts something happens "after 100ms" passes locally and fails on the slow CI runner with a tick that arrived a millisecond late. Root cause. `time.Sleep` and `time.NewTimer` use the real clock. A test that sleeps 100ms and asserts a side effect happened is racing the scheduler. Locally the spare CPU finishes in 102ms; on CI the throttled runner finishes in 105ms and the assertion fires before the side effect lands. ``` func TestRefreshFiresEvery100ms(t *testing.T) { count := 0 stop := startRefresher(func() { count++ }, 100*time.Millisecond) defer stop() time.Sleep(250 * time.Millisecond) if count < 2 { t.Fatalf("expected at least 2 ticks, got %d", count) // sometimes 1 on CI } } ``` Fix. Use [`testing/synctest`](https://pkg.go.dev/testing/synctest) (Go 1.24+) to virtualize time inside the test: time advances when every goroutine in the bubble is blocked on something time-dependent, so a 100ms sleep is instant and exact. For older Go, inject a `Clock` interface and substitute a fake. ``` func TestRefreshFiresEvery100ms(t *testing.T) { synctest.Run(func() { count := 0 stop := startRefresher(func() { count++ }, 100*time.Millisecond) defer stop() time.Sleep(250 * time.Millisecond) // virtual time, deterministic if count != 2 { t.Fatalf("expected exactly 2 ticks, got %d", count) } }) } ``` With Mergify. Test Engine links the failure to its CI runner type. When a timing test only fails on the slower runner pool and never on the laptop pool, the dashboard surfaces the resource sensitivity so the real-clock dependency is the obvious culprit. Pattern 5 ### httptest.Server connection leaks Symptom. A long-running test suite eventually fails with \`dial tcp: too many open files\` or a flaky test panics with \`address already in use\`. Root cause. [`httptest.NewServer`](https://pkg.go.dev/net/http/httptest#NewServer) binds a port and starts a goroutine. Forgetting to call `srv.Close()` leaks both. After enough tests, the worker hits the file-descriptor limit. The first test to run after that point fails with an unrelated socket error. ``` func TestUploadHandler(t *testing.T) { srv := httptest.NewServer(uploadHandler()) // no defer srv.Close() resp, _ := http.Post(srv.URL, "application/octet-stream", strings.NewReader("hi")) if resp.StatusCode != 200 { t.Fatalf("got %d", resp.StatusCode) } } ``` Fix. Always defer `srv.Close()`, or register it with `t.Cleanup` so the cleanup survives early returns. Pair with `resp.Body.Close()` for every HTTP response to release the underlying connection. ``` func TestUploadHandler(t *testing.T) { srv := httptest.NewServer(uploadHandler()) t.Cleanup(srv.Close) resp, err := http.Post(srv.URL, "application/octet-stream", strings.NewReader("hi")) if err != nil { t.Fatal(err) } defer resp.Body.Close() if resp.StatusCode != 200 { t.Fatalf("got %d", resp.StatusCode) } } ``` With Mergify. Test Engine groups failures whose only signature is \`too many open files\` or \`address already in use\` into a single fd-exhaustion bucket. The dashboard surfaces the test that first triggered the leak rather than the random victim that ran when fds ran out. Pattern 6 ### TestMain teardown bypassed by os.Exit Symptom. A test suite that uses TestMain to spin up a database container leaves the container running after a failure, and the next CI run fails with \`port already allocated\`. Root cause. [`TestMain`](https://pkg.go.dev/testing#hdr-Main) patterns that defer teardown and then call `os.Exit(code)` never run the deferred function: `os.Exit` terminates the process without unwinding the stack. The same trap shows up for code under test that calls `log.Fatal` or `os.Exit` directly, or for a goroutine that panics outside any test function and crashes the whole process before m.Run() returns. ``` func TestMain(m *testing.M) { pool := startTestDB() defer pool.Close() // SKIPPED: os.Exit never runs deferred funcs os.Exit(m.Run()) } ``` Fix. Move `os.Exit` into a wrapper that returns the exit code from a function whose deferred teardown runs first. Inside test bodies, replace `log.Fatal` with `t.Fatal` so the runner can clean up; for goroutines started in tests, recover and report through `t.Errorf` instead of letting the panic crash the process. ``` func TestMain(m *testing.M) { os.Exit(run(m)) } func run(m *testing.M) int { pool := startTestDB() defer pool.Close() // runs before os.Exit return m.Run() } ``` With Mergify. Test Engine notices that the first test of a CI run fails for the same \`port already allocated\` reason and the failure correlates with a panic on the previous run. The dashboard surfaces the cross-run signature so the missing teardown is easy to spot. Pattern 7 ### Subtest plus t.Parallel interleaving surprises Symptom. A table-driven test with \`t.Parallel()\` inside the subtest passes locally and fails on CI with subtests sharing the same loop-variable state. Root cause. `t.Run` with `t.Parallel()` inside delays the subtest until the parent finishes its body, then runs all parallel subtests together. Pre-Go 1.22 this combined with [loop variable capture](https://go.dev/blog/loopvar-preview) to give every subtest the same value of the loop variable. Go 1.22+ fixed the loop-var semantics but the parent-finishes-first ordering still surprises people who assume subtests run in declaration order. ``` func TestPricing(t *testing.T) { cases := []struct{ plan string; want int }{ {"free", 0}, {"pro", 99}, {"enterprise", 999}, } for _, tc := range cases { t.Run(tc.plan, func(t *testing.T) { t.Parallel() got := priceFor(tc.plan) if got != tc.want { // pre-Go 1.22: tc was the last loop value for every subtest t.Fatalf("plan %q: want %d, got %d", tc.plan, tc.want, got) } }) } } ``` Fix. Stay on Go 1.22 or newer where loop variables are scoped per iteration. If you cannot upgrade, capture the loop variable explicitly (`tc := tc`) before the subtest body. For tests that need to share setup across subtests, run setup in the parent and pass it explicitly through closure. ``` func TestPricing(t *testing.T) { cases := []struct{ plan string; want int }{ {"free", 0}, {"pro", 99}, {"enterprise", 999}, } for _, tc := range cases { // Go 1.22+: tc is per-iteration, no manual shadow needed t.Run(tc.plan, func(t *testing.T) { t.Parallel() if got := priceFor(tc.plan); got != tc.want { t.Fatalf("plan %q: want %d, got %d", tc.plan, tc.want, got) } }) } } ``` With Mergify. Test Engine groups failing subtests by their parent. When every subtest of TestPricing fails with the same wrong value, the dashboard surfaces the loop-variable signature so the upgrade or shadow fix lands once. Pattern 8 ### CI retries hiding flaky Go test failures Symptom. Your first CI run fails, a retry passes, and the pipeline reports green. A user still hits the race in production. Root cause. `go test -count=N` reruns tests N times and exits non-zero if any run fails, so it tends to expose flakes rather than mask them. The masking happens when CI or shell scripts retry the whole suite and only keep the final result. That turns "failed once, passed later" into "green" even though the underlying race or order dependency is still there. Cached results are a separate concern; use `-count=1` when you need a fresh run. ``` # CI script (please don't) go test ./... || go test ./... || go test ./... ``` Fix. Do not hide failures behind blind suite retries. Let the first failing attempt stay visible, then fix the flaky test or quarantine it explicitly while you work on the root cause. If you need a fresh run instead of cached results, use `go test -count=1` without treating later retries as success. With Mergify. Test Engine reruns at the CI level with attempt-by-attempt result tracking. You can see that a test failed on attempt 1 and passed on attempt 2, which is exactly the signal that naive shell retries would hide. Quarantine kicks in once the pattern is clear. Detection ## How to detect flaky Go tests in CI Add gotestsum (or any JUnit-emitting Go test runner) to your CI script, point it at your packages, and upload the resulting XML to Mergify with a one-line CLI call. Test Engine builds a confidence score for every test on your default branch. PR runs are compared against that baseline. Anything inconsistent gets flagged in a PR comment before the author merges. mergify ci ``` # 1. Install gotestsum (one-time) go install gotest.tools/gotestsum@latest # 2. Emit JUnit XML on every CI run gotestsum --junitfile junit.xml -- -race -shuffle=on ./... # 3. Upload the result (once, in CI) curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh mergify ci junit-process junit.xml ``` Prevention ## How to prevent flaky Go tests from merging On every PR, Mergify reruns the tests whose confidence is below threshold, without \`go test -count=N\` touching your config. The PR gets a comment naming the unreliable tests, their confidence history, and whether the failure on this PR is new or historical noise. Authors fix the real bugs before merge instead of re-running CI until it passes. Quarantine ## How to quarantine flaky Go tests without skipping them Once a Go test is confirmed flaky, Test Engine quarantines it. The test still runs in the suite, no \`t.Skip()\` rewrite required, but its result no longer blocks merges or marks the pipeline red. When the pass rate on main recovers, quarantine lifts automatically and the test goes back to being load-bearing. Your CI ## Run it on your CI The Go setup for Test Engine on each CI, how its retries compare with the CI's own reruns, and what makes Go flake on that CI's runners. - [Go on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions/go) ## Want to see which Go tests in your repo are already flaky? Works with gotestsum or any JUnit-compatible Go test runner. Setup takes under five minutes. [Book a discovery call](https://mergify.com/discovery) ## Frequently asked questions Why are my Go tests flaky in CI but pass locally? Your laptop and your CI runner differ in CPU count, parallelism, and the random seed Go uses for map iteration and goroutine scheduling. Tests that race on package-level state, leak goroutines into the next test, or assume a map iteration order surface those issues only under CI's tighter resource budget. Reproduce locally with \`go test -race -count=10 -shuffle=on ./...\` and let the race detector and shuffler push the failure into the open before you push. How do I detect flaky Go tests? Go alone cannot tell flaky from broken since each run gives one data point per test. You need to run the same commit multiple times and compare results. Mergify Test Engine does that on every PR and on the default branch, scores each test, and surfaces the tests whose pass rate drops below a confidence threshold. Does \`go test -count=N\` fix flaky tests? No, it hides them. A test that fails on attempt 1 and passes on attempt 2 is still broken; you have only decided not to look at the failure. Use \`go test -count=N\` as a temporary bandage for a test you are actively fixing, never as a permanent policy. For visibility without blocking the merge queue, quarantine instead of retry. What causes flaky tests in Go? Eight patterns cover most of what we see: t.Parallel() racing on shared package state, goroutine leaks across test boundaries, map iteration order assumptions, time-based assertions without testing/synctest, httptest.Server connection leaks, TestMain teardown bypassed by os.Exit, subtest plus t.Parallel interleaving surprises, and CI retries hiding flaky Go test failures. Each is covered above with a minimal reproducer. How do I quarantine a flaky Go test without deleting it? Mergify Test Engine quarantines the test automatically once its confidence score drops. The test still runs in the suite, but a failing result no longer blocks merges and its noise no longer drowns out real signal. When the test stabilizes on main, quarantine lifts automatically. No \`t.Skip()\`, no commented-out tests, no orphaned files. Why does my Go test pass with -parallel=1 but fail without it? Because \`t.Parallel()\` runs that test alongside any other parallel test in the same package, and they share package-level state by default. A package-level \`var\`, an \`init()\`-time singleton, an environment variable, or a shared external resource (a file, a port, a database row) becomes a race the moment two parallel tests touch it. Move the state inside the test body (\`t.TempDir()\`, \`t.Setenv()\`, a per-test row scoped by \`t.Name()\`) and the parallel mode stops mattering. ## Ship your Go suite green. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/jest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/Jest # Flaky tests in Jest. Named, fixed, and quarantined. Flaky Jest suites are not random. They follow patterns: fake-timer leakage, snapshot drift, mock hoisting, concurrent state. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Updated April 2026 Example PR comment from the Mergify bot detecting a flaky Jest test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky Jest suite: fake-timer leakage, snapshot drift on shared state, module-mock hoisting, test.concurrent state sharing, unhandled promise leaks, retryTimes hiding real bugs, environment bleed between node and jsdom, and globalSetup hooks that forgot to await. Each one has a minimal reproducer, a fix in stock Jest, and a Mergify Test Engine detection strategy below. ## Why are Jest tests flaky? Jest ships a lot of useful machinery: fake timers as a first-class primitive, [automatic mock hoisting](https://jestjs.io/docs/jest-object#jestmockmodulename-factory-options), snapshot serialization, and a choice of two default environments (node or jsdom). Every one of those features pays for itself. Every one of them is also a flake surface if you lean on it without cleaning up after yourself. The common thread is implicit shared state. Timers, mocks, module caches, the `window` object, snapshot files: they all persist across tests by design, and Jest's test-file-level parallelism means stale state from one file can leak into the next worker. That is where most "passes locally, fails in CI" stories come from. Not randomness, not the network, not the moon phase. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of Jest suites: fake-timer leakage, snapshot drift on shared state, module-mock hoisting traps, test.concurrent state sharing, unhandled promise leaks, jest.retryTimes() hiding real bugs, environment bleed between node and jsdom, and globalSetup hooks that forgot to await. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01Fake-timer leakage](#pattern-fake-timer-leakage) 2. [02Snapshot drift on shared state](#pattern-snapshot-drift-on-shared-state) 3. [03Module-mock hoisting traps](#pattern-module-mock-hoisting-traps) 4. [04test.concurrent state sharing](#pattern-test-concurrent-state-sharing) 5. [05Unhandled promise leaks](#pattern-unhandled-promise-leaks) 6. [06Retry-hidden failures](#pattern-retry-hidden-failures) 7. [07Environment bleed](#pattern-environment-bleed) 8. [08globalSetup hook races](#pattern-globalsetup-hook-races) Pattern 1 ### Fake-timer leakage Symptom. A test passes in isolation, fails when the suite runs end to end, and the failure looks like a completely unrelated assertion timeout. Root cause. Calling [jest.useFakeTimers()](https://jestjs.io/docs/timer-mocks) mutates a module-scoped timer mock that survives the test unless you explicitly restore it. setTimeout/setInterval callbacks queued under fake timers can fire under the next test's real timers, and vice versa. The next test sees phantom assertion runs, mock calls it did not make, or promise states that make no sense. ``` test("fake-timer leakage", () => { jest.useFakeTimers(); setTimeout(() => fetchUser(), 5000); // no cleanup. fake timers leak into the next test }); test("getUser returns the logged-in user", async () => { const user = await getUser(); expect(user).toBeDefined(); // fetchUser from test 1 fires here }); ``` Fix. Pair every `useFakeTimers()` with `useRealTimers()` in `afterEach`, and clear pending timers before the switch. If you use fake timers everywhere, move it into a global setup and stop flipping per-test. ``` afterEach(() => { jest.clearAllTimers(); jest.useRealTimers(); }); ``` With Mergify. Test Engine reruns the suspect test on a dedicated worker. When the same SHA produces different results across runs, the test is flagged; when two of the three runs agree, the dissenter is quarantined so the merge queue keeps moving. Pattern 2 ### Snapshot drift on shared state Symptom. A snapshot test fails every few days with a diff you did not cause: a timestamp shifted, a UUID changed, a list reordered. Root cause. `toMatchSnapshot()` serializes whatever you hand it. If the rendered output includes `Date.now()`, a crypto UUID, a `Map` iterated in insertion order, or a test-id from your component library, the snapshot captures today's value and tomorrow's run disagrees. The test has not changed. The world has. ``` test("renders invoice", () => { // Invoice renders 'Generated: {Date.now()}' and a random receipt UUID const tree = renderer.create().toJSON(); expect(tree).toMatchSnapshot(); }); ``` Fix. Freeze the sources of non-determinism with a custom serializer or a `jest.setSystemTime()` stub. For UUIDs, mock the generator. For anything truly variable, assert on structure with [property matchers](https://jestjs.io/docs/snapshot-testing#property-matchers) rather than a full snapshot. ``` expect.addSnapshotSerializer({ test: (v) => typeof v === "string" && UUID_RE.test(v), print: () => '""', }); beforeAll(() => { jest.useFakeTimers().setSystemTime(new Date("2026-01-01")); }); ``` With Mergify. Test Engine groups the repeated diffs under a single flaky-test confidence score so you see the pattern at once: same test, same file, three drifting snapshots in a week. Quarantine until you add the serializer. Pattern 3 ### Module-mock hoisting traps Symptom. A mock you thought you scoped to one file starts affecting a later file in the same worker, or your imported module is undefined on the second run. Root cause. `jest.mock('./db')` is hoisted above imports at transform time. That works great when the factory is pure, but if the factory captures a closure variable or your mock caches state, the second test file that imports `./db` gets the cached mock, not a fresh one. Jest workers reuse module caches across files by default. ``` // auth.test.ts jest.mock("./db", () => ({ query: jest.fn().mockResolvedValue([]) })); import { login } from "./auth"; test("login calls db.query", async () => { /* ... */ }); // billing.test.ts, same worker import { charge } from "./billing"; test("charge hits the db", async () => { await charge(); // expected: real db.query. actual: the mock from auth.test.ts }); ``` Fix. Reset modules between files with `jest.resetModules()`, or set [`resetMocks: true`](https://jestjs.io/docs/configuration#resetmocks-boolean) in your config so every test starts clean. If a mock is truly global, put it in `__mocks__` where the behavior is explicit. ``` // jest.config.js module.exports = { resetMocks: true, resetModules: true, }; ``` With Mergify. Test Engine detects the cross-file signature: a test that only fails when run after a specific other file. The dashboard surfaces the ordering dependency so you know where to look. Pattern 4 ### test.concurrent state sharing Symptom. A suite written with test.concurrent fails intermittently with off-by-one counters, wrong user records, or assertions that see state they should not. Root cause. [`test.concurrent`](https://jestjs.io/docs/api#testconcurrentname-fn-timeout) runs tests in parallel inside a single file using Jest's promise scheduler. Any module-level variable they touch is shared. `beforeEach` reset hooks do not fire between concurrent tests. They fire before each of them, in order, which is not the same thing. ``` let counter = 0; beforeEach(() => { counter = 0; }); test.concurrent("increments to 1", async () => { counter++; expect(counter).toBe(1); // passes in isolation }); test.concurrent("increments to 1 again", async () => { counter++; expect(counter).toBe(1); // fails ~50% of the time }); ``` Fix. Only use `test.concurrent` for tests that are pure functions of their inputs. If a test reaches for a shared mock, a database, or a module singleton, demote it to a regular `test`. With Mergify. Test Engine surfaces concurrent-only failures distinctly: the test fails only when its file runs with --maxConcurrency > 1, and passes in sequential reruns. That signature is the fingerprint of shared-state parallelism, and the dashboard labels it. Pattern 5 ### Unhandled promise leaks Symptom. A test completes green, then a later test fails with an error message that mentions something the earlier test was doing. Root cause. A test that calls `.then()` without [returning or awaiting](https://jestjs.io/docs/asynchronous) the promise finishes before the promise resolves. The resolution lands in the next test's microtask queue, where it can mutate shared state, call a mock that now belongs to a different test, or trigger an `unhandledRejection` that Jest attributes to the wrong test. ``` test("fetches the user", () => { fetchUser().then((u) => { // this resolves after the test body has already returned expect(u.name).toBe("Rémy"); }); }); ``` Fix. Always `await` or `return` promises from test bodies. Turn on the `@typescript-eslint/no-floating-promises` rule to catch this in review. ``` test("fetches the user", async () => { const u = await fetchUser(); expect(u.name).toBe("Rémy"); }); ``` With Mergify. Test Engine reruns the failing test on a worker by itself. If the test passes alone but fails under the original schedule, the dashboard tags it with a 'sequencing-dependent' label and suggests the floating-promise pattern. Pattern 6 ### Retry-hidden failures Symptom. Your pipeline is green. Your users hit a bug in production that your tests were supposed to catch. Root cause. [`jest.retryTimes(n)`](https://jestjs.io/docs/jest-object#jestretrytimesnumretries-options) re-runs a failing test up to n times and reports the last result. A real bug that fails on the first attempt and passes on the second (because of caching, warm-up, or a race it happens to lose locally) gets reported as green. Retries at the Jest level are a tempting knob; they paper over the exact kinds of failures that CI is supposed to catch. ``` // jest.config.js (please don't) module.exports = { setupFilesAfterEach: ["/jest.setup.js"], }; // jest.setup.js jest.retryTimes(3, { logErrorsBeforeRetry: false }); ``` Fix. Do not retry at the Jest level. When a test is genuinely flaky, diagnose and fix it. When diagnosis takes more than a session, quarantine it instead. That keeps the signal visible without blocking merges. With Mergify. Test Engine reruns at the CI level with attempt-level result tracking. You see that a test passed on attempt 2 of 3, which is exactly the information `retryTimes()` throws away. Quarantine kicks in once the pattern is clear. Pattern 7 ### Environment bleed Symptom. A test passes in your file but throws ReferenceError: document is not defined in another file that tests the same component. Root cause. Jest lets you choose between the `node` and `jsdom` environments per file with a [`@jest-environment` pragma](https://jestjs.io/docs/configuration#testenvironment-string). Forget the pragma on a file that needs DOM globals and the test throws on load. Keep the pragma on a file that no longer needs DOM and you pay the jsdom startup cost on every run for no reason. ``` // ComponentA.test.tsx: has the pragma, works /** * @jest-environment jsdom */ import { render } from "@testing-library/react"; // ComponentB.test.tsx: forgot the pragma import { render } from "@testing-library/react"; test("renders", () => { render(); // ReferenceError: document is not defined }); ``` Fix. Set `testEnvironment: "jsdom"` in `jest.config.js` for your UI test directory via `projects`, and leave `node` as the default for everything else. Per-file pragmas should be the exception, not the rule. ``` // jest.config.js module.exports = { projects: [ { displayName: "unit", testEnvironment: "node", testMatch: ["/src/**/*.test.ts"] }, { displayName: "ui", testEnvironment: "jsdom", testMatch: ["/src/**/*.test.tsx"] }, ], }; ``` With Mergify. Environment misconfigurations manifest as consistent failures when the file runs alone, not as intermittent flakes. Test Engine still catches them through confidence scoring on the default branch, surfacing the file as broken rather than flaky. Pattern 8 ### globalSetup hook races Symptom. The first test of a new CI run fails with a 'table does not exist' or 'fixture not found' error. A rerun of the same SHA passes. Root cause. [`globalSetup`](https://jestjs.io/docs/configuration#globalsetup-string) runs once before the suite. If its function returns before the async work it kicked off is done, Jest starts your tests against a half-seeded database or a half-built temp directory. The first few tests fail; by the time the setup actually completes, later tests see a consistent world. ``` // setup.ts: the bug export default function () { seedDatabase(); // returns a promise, never awaited } ``` Fix. Make the setup function `async` and await every side effect. Jest blocks until the returned promise resolves. ``` export default async function () { await seedDatabase(); } ``` With Mergify. Test Engine notices that the first N tests of a run fail consistently only on cold CI workers, which is the fingerprint of a setup race. The dashboard flags the ordering and the subset of affected tests. Detection ## How to detect flaky Jest tests in CI Point Jest at the jest-junit reporter, upload the resulting XML to Mergify with a one-line CLI call, and Test Engine builds a confidence score for every test on your default branch. PR runs are compared against that baseline. Anything inconsistent gets flagged in a PR comment before the author merges. mergify ci ``` # 1. Add the jest-junit reporter npm install --save-dev jest-junit # 2. Emit JUnit XML on every CI run jest --reporters=default --reporters=jest-junit # 3. Upload the result (once, in CI) curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh mergify ci junit-process junit.xml ``` Prevention ## How to prevent flaky Jest tests from merging On every PR, Mergify reruns the tests whose confidence is below threshold, without jest.retryTimes() touching your config. The PR gets a comment naming the unreliable tests, their confidence history, and whether the failure on this PR is new or historical noise. Authors fix the real bugs before merge instead of re-running CI until it passes. Quarantine ## How to quarantine flaky Jest tests without skipping them Once a Jest test is confirmed flaky, Test Engine quarantines it. The test still runs in the suite, no \`describe.skip()\` rewrite required, but its result no longer blocks merges or marks the pipeline red. When the pass rate on main recovers, quarantine lifts automatically and the test goes back to being load-bearing. Your CI ## Run it on your CI The Jest setup for Test Engine on each CI, how its retries compare with the CI's own reruns, and what makes Jest flake on that CI's runners. - [Jest on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions/jest) ## Want to see which Jest tests in your repo are already flaky? Works with jest-junit, jest-github-actions-reporter, or any JUnit-compatible Jest reporter. Setup takes under five minutes. [Book a discovery call](https://mergify.com/discovery) ## Frequently asked questions Why are my Jest tests flaky in CI but pass locally? Your laptop and your CI runner differ in CPU, parallelism, and timing. Tests that use real timers, touch the filesystem, or forget to await a promise lose the race under CI's tighter budget and hide on your laptop. Run the suite with \`jest --runInBand --no-cache\` locally to approximate CI behavior, then fix the underlying timing or isolation bug before you push. How do I detect flaky Jest tests? Jest alone cannot tell flaky from broken since each run gives one data point per test. You need to run the same commit multiple times and compare results. Mergify Test Engine does that on every PR and on the default branch, scores each test, and surfaces the tests whose pass rate drops below a confidence threshold. Does jest.retryTimes() fix flaky tests? No, it hides them. A test that fails on attempt 1 and passes on attempt 2 is still broken; you have only decided not to look at the failure. Use jest.retryTimes() as a temporary bandage for a test you are actively fixing, never as a permanent policy. For visibility without blocking the merge queue, quarantine instead of retry. What causes flaky tests in Jest? Eight patterns cover most of what we see: fake-timer leakage, snapshot drift on shared state, module-mock hoisting traps, test.concurrent state sharing, unhandled promise leaks, jest.retryTimes() hiding real bugs, environment bleed between node and jsdom, and globalSetup hooks that forgot to await. Each is covered above with a minimal reproducer. How do I quarantine a flaky Jest test without deleting it? Mergify Test Engine quarantines the test automatically once its confidence score drops. The test still runs in the suite, but a failing result no longer blocks merges and its noise no longer drowns out real signal. When the test stabilizes on main, quarantine lifts automatically. No \`test.skip()\`, no commented-out tests, no orphaned files. Is test.concurrent flaky by default? No, but it fails predictably on any shared module-level state: counters, caches, singletons, spies. \`beforeEach\` does not fire between concurrent tests (it fires before each of them, in order), so reset hooks do not protect concurrent state. Reserve \`test.concurrent\` for pure tests and demote anything that touches shared state to a regular \`test\`. Keep digging ## Deep dives on the patterns above [All posts →](https://mergify.com/blog) [ ](https://mergify.com/blog/fake-timer-leakage-in-jest) April 25, 2026·6 min read [ ### Fake-timer leakage in Jest: the flake nobody sees coming ](https://mergify.com/blog/fake-timer-leakage-in-jest) Why a single jest.useFakeTimers() call can pollute the next test in the file, what the failure looks like, and the three-line afterEach that makes it go away for good. Rémy Duthu [ ](https://mergify.com/blog/jest-snapshot-drift-shared-state) April 25, 2026·7 min read [ ### Jest snapshot drift: when toMatchSnapshot lies about your code ](https://mergify.com/blog/jest-snapshot-drift-shared-state) Snapshots fail every few days with diffs you did not cause. Most teams blame their Date.now() mock. The real story involves three less-obvious sources of drift, and a one-line serializer that fixes them. Rémy Duthu [ ](https://mergify.com/blog/why-jest-retrytimes-hides-bugs) April 25, 2026·5 min read [ ### Why jest.retryTimes() is hiding bugs in your test suite ](https://mergify.com/blog/why-jest-retrytimes-hides-bugs) Adding retryTimes(3) feels like fixing flakes. It is not. It is throwing away the data you need to find the bug, then telling CI everything is fine. Rémy Duthu ## Ship your Jest suite green. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/junit [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/JUnit # Flaky tests in JUnit. Named, fixed, and quarantined. Flaky JUnit suites are not random. They follow patterns: static state leaking across forks, Spring context cache evictions, Mockito mock bleed, real-clock assertions, leaked MockWebServers. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Published April 2026 Example PR comment from the Mergify bot detecting a flaky JUnit test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky JUnit suite: method-order assumptions without @TestMethodOrder, static-field state leaking across tests, Surefire forkCount and parallel resources races, Mockito mock leakage under @TestInstance(PER\_CLASS), @SpringBootTest context-cache surprises, time-based assertions without an injected Clock, MockWebServer instances that never get shut down, and Surefire rerunFailingTestsCount hiding real bugs. Each one has a minimal reproducer, a fix in stock JUnit, and a Mergify Test Engine detection strategy below. ## Why are JUnit tests flaky? JUnit 5 is the assembly language of Java testing: a small core, a large extension surface, and a default model that runs every test against a single JVM unless you tell it otherwise. The single-JVM default is the appeal (cold start matters when you have ten thousand tests) and the source of most flakes. `static` fields, the Spring context, the system clock, the system class loader, and any singleton that lives in static state survives across every test in the same fork. Layer on the build tooling. [Maven Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) and Gradle's test runner add forking and parallelism on top of JUnit, and a parallel run that worked sequentially can race the moment two tests in the same fork touch the same static cache. `@MockBean` in a Spring Boot test forces a context reload that breaks adjacent tests still expecting the cached context. Each pattern has a clean fix once you can name the failure mode. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of JUnit suites: method-order assumptions without @TestMethodOrder, static-field state leaking across tests, Surefire forkCount and parallel resources races, Mockito mock leakage under @TestInstance(PER\_CLASS), @SpringBootTest context-cache surprises, time-based assertions without an injected Clock, MockWebServer instances that never get shut down, and Surefire rerunFailingTestsCount hiding real bugs. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01Method-order assumptions without @TestMethodOrder](#pattern-method-order-assumptions-without-testmethodorder) 2. [02Static-field state leaking across tests](#pattern-static-field-state-leaking-across-tests) 3. [03Surefire forkCount and parallel resources races](#pattern-surefire-forkcount-and-parallel-resources-races) 4. [04Mockito mock leakage under PER\_CLASS lifecycle](#pattern-mockito-mock-leakage-under-per-class-lifecycle) 5. [05@SpringBootTest context-cache surprises](#pattern-springboottest-context-cache-surprises) 6. [06Time-based assertions without an injected Clock](#pattern-time-based-assertions-without-an-injected-clock) 7. [07MockWebServer instances that never get shut down](#pattern-mockwebserver-instances-that-never-get-shut-down) 8. [08Surefire rerunFailingTestsCount hiding real bugs](#pattern-surefire-rerunfailingtestscount-hiding-real-bugs) Pattern 1 ### Method-order assumptions without @TestMethodOrder Symptom. A test class that has been green for years suddenly fails on a CI runner with a different JDK build, with one test asserting on state another test creates. Root cause. [JUnit 5 does not guarantee test method execution order by default](https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-execution-order). The platform picks an order at runtime that is deterministic but not promised across JDK versions, JVM startup options, or platform updates. A test class that implicitly depends on test ordering (test A creates a record, test B reads it) keeps passing until the order changes underneath you. ``` class UserServiceTest { private static User created; @Test void createsUser() { created = userService.create("Rémy"); assertNotNull(created); } @Test void readsCreatedUser() { // assumes createsUser ran first; brittle User found = userService.findById(created.getId()); assertEquals("Rémy", found.getName()); } } ``` Fix. Make every test independent. Build the prerequisite state in `@BeforeEach` or inside the test itself. When a test really must run after another (database migration smoke tests, end-to-end happy paths), declare it explicitly with [`@TestMethodOrder(OrderAnnotation.class)`](https://junit.org/junit5/docs/current/api/org.junit.jupiter.api/org/junit/jupiter/api/MethodOrderer.html) and number every method. ``` class UserServiceTest { @Test void createsUser() { User created = userService.create("Rémy"); assertNotNull(created); } @Test void readsCreatedUser() { User created = userService.create("Rémy"); User found = userService.findById(created.getId()); assertEquals("Rémy", found.getName()); } } ``` With Mergify. Test Engine records the JVM and JDK version of every CI run and groups failures by their environment fingerprint. When a test class only fails after a JDK upgrade and the failures all sit in the same class, the dashboard surfaces the order-dependence pattern at PR time. Pattern 2 ### Static-field state leaking across tests Symptom. A test that mutates a static counter or a feature-flag map passes alone and fails when its class shares a Surefire fork with another test class. Root cause. JUnit creates a fresh instance of the test class for each test method by default, but `static` fields live for the JVM. A static cache, a static "current user" set in test A, or a flag toggled in `@BeforeAll` survives every test that runs after, including in unrelated test classes that share the fork. ``` class FeatureFlagTest { static Map FLAGS = new HashMap<>(); @BeforeAll static void setUp() { FLAGS.put("NEW_BILLING", true); } @Test void newBillingPathFires() { assertTrue(FLAGS.get("NEW_BILLING")); } } // elsewhere in the same fork class LegacyBillingTest { @Test void legacyChargeReturnsCorrectAmount() { // expected: NEW_BILLING == false (default). Actual: true, leaked. assertEquals(100, billingService.charge(...)); } } ``` Fix. Move shared state into instance fields and reset it in `@AfterEach`, or use [`@TestInstance(PER_CLASS)`](https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-instance-lifecycle) with explicit cleanup. For genuine global state (system properties, time zones), use a JUnit extension that snapshots and restores around each test. ``` class FeatureFlagTest { private FeatureFlags flags; @BeforeEach void setUp() { flags = new FeatureFlags(); flags.set("NEW_BILLING", true); } @Test void newBillingPathFires() { assertTrue(flags.get("NEW_BILLING")); } } ``` With Mergify. Test Engine detects the cross-class signature: test class B fails consistently when class A ran in the same fork and never alone. The dashboard groups failures by their predecessor so the leaking-static pattern is the obvious lead. Pattern 3 ### Surefire forkCount and parallel resources races Symptom. A test suite that runs green with Surefire \`-T 1\` fails with \`-T 4\` on assertions involving counters, ports, or temp directory paths. Root cause. [Surefire's `forkCount` and parallel execution](https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html) run multiple tests in the same JVM concurrently when configured with parallel mode. Anything those threads share (a static counter, an in-memory database with a fixed name, a hardcoded port) becomes a race the moment parallelism kicks in. The test does not change; the build configuration does. ``` maven-surefire-plugin methods 4 1 // CounterTest.java class CounterTest { static AtomicInteger counter = new AtomicInteger(); @Test void incrementsToOne() { int v = counter.incrementAndGet(); assertEquals(1, v); // fails when 4 threads share the same counter } } ``` Fix. Pick a parallelism mode that matches the suite's isolation guarantees. [JUnit 5's per-class concurrency](https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution) with one thread per class avoids method-level interleaving while still parallelizing across classes. For shared external resources (ports, files, database schemas), use unique names per test (`@TempDir`, ephemeral ports, namespace by `TestInfo`). ``` # junit-platform.properties junit.jupiter.execution.parallel.enabled = true junit.jupiter.execution.parallel.mode.default = same_thread junit.jupiter.execution.parallel.mode.classes.default = concurrent ``` With Mergify. Test Engine tags failures that only appear under parallel runs as parallelism-sensitive and never under sequential reruns. The dashboard surfaces the parallel-only signature so you know the test is not broken on its own. Pattern 4 ### Mockito mock leakage under PER\_CLASS lifecycle Symptom. A spy or mock you set up in one test starts returning canned values in a sibling test that never declared it, and \`@BeforeEach\` resets do not help. Root cause. JUnit 5's default lifecycle creates a fresh test instance per method, so `@Mock` fields are recreated and stubbing does not leak. Switch to [`@TestInstance(Lifecycle.PER_CLASS)`](https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-instance-lifecycle) (often added when a class needs a non-static `@BeforeAll` or to amortize setup) and the same instance is shared across every method. `@Mock` fields, the `@InjectMocks` subject, and any stubbing you set in one test all carry over into the next. ``` @ExtendWith(MockitoExtension.class) @TestInstance(Lifecycle.PER_CLASS) // one instance for the whole class class UserServiceTest { @Mock UserRepository repo; @InjectMocks UserService service; @Test void findsUserById() { when(repo.findById(1L)).thenReturn(Optional.of(new User("Rémy"))); assertEquals("Rémy", service.find(1L).getName()); } @Test void returnsEmptyWhenNotFound() { // repo.findById(1L) still returns the stub from the previous test // because the instance (and its mocks) is shared under PER_CLASS assertTrue(service.find(2L).isEmpty()); } } ``` Fix. Default to JUnit 5's per-method lifecycle so mocks are recreated per test. When you need `@TestInstance(PER_CLASS)` for non-static `@BeforeAll` hooks, reset mocks explicitly in `@AfterEach` with `Mockito.reset(...)` so stubbing does not carry across tests. ``` @ExtendWith(MockitoExtension.class) @TestInstance(Lifecycle.PER_CLASS) class UserServiceTest { @Mock UserRepository repo; @InjectMocks UserService service; @AfterEach void resetMocks() { Mockito.reset(repo); // clear stubbing between tests } @Test void findsUserById() { when(repo.findById(1L)).thenReturn(Optional.of(new User("Rémy"))); assertEquals("Rémy", service.find(1L).getName()); } @Test void returnsEmptyWhenNotFound() { assertTrue(service.find(2L).isEmpty()); } } ``` With Mergify. Test Engine groups failures whose only signature is \`UnnecessaryStubbingException\` or assertion-on-stubbed-default into a single mock-leakage bucket. The dashboard surfaces the offending test method so the missing reset is one fix instead of N. Pattern 5 ### @SpringBootTest context-cache surprises Symptom. A Spring Boot integration test passes alone, fails when the suite runs end to end with \`Failed to load ApplicationContext\`, and a rerun of the failing class passes. Root cause. [Spring caches `ApplicationContext` instances](https://docs.spring.io/spring-framework/reference/testing/testcontext-framework/ctx-management/caching.html) by their configuration signature. `@MockBean`, `@DynamicPropertySource`, and `@TestPropertySource` change that signature, forcing a context reload that disposes the old context. A test that holds a reference to a bean from the cached context fails with a `BeanCreationException` when a sibling triggers a reload. ``` @SpringBootTest class CheckoutTest { @MockBean PaymentClient payments; // forces a fresh context @Test void chargesCard() { ... } } @SpringBootTest class InventoryTest { @Autowired InventoryService inventory; // expected to be from the cached context @Test void decrementsStock() { // CheckoutTest's @MockBean evicted the cached context // inventory was wired against the now-disposed context → fails } } ``` Fix. Group tests that share configuration into a base class so they reuse one context. When you genuinely need a different bean shape, accept the reload cost rather than scattering `@MockBean` across many test classes. [Spring's context cache statistics](https://docs.spring.io/spring-framework/reference/testing/testcontext-framework/ctx-management/caching.html#testcontext-ctx-management-caching) in the test log show how many reloads your suite triggers. ``` // Shared base class @SpringBootTest abstract class IntegrationTestBase { @MockBean PaymentClient payments; // declared once } class CheckoutTest extends IntegrationTestBase { ... } class InventoryTest extends IntegrationTestBase { ... } ``` With Mergify. Test Engine catches the order-dependent signature: a Spring test only fails when run after a specific other test class that shifts the context configuration. The dashboard tags the dependency so the @MockBean blast radius is visible. Pattern 6 ### Time-based assertions without an injected Clock Symptom. A test that asserts an event happens "within the last second" passes locally and fails on the slower CI runner with a timestamp 1.2 seconds old. Root cause. Calling `Instant.now()` or `LocalDateTime.now()` directly inside production code reads the system clock. A test that asserts on a freshly created timestamp races against scheduler jitter. Locally the assertion fires within a millisecond; on CI the same code runs after a longer pause and the assertion's tolerance window is too tight. ``` class AuditLogger { void log(String event) { store.put(event, Instant.now()); // direct clock read } } @Test void auditTimestampIsRecent() { auditLogger.log("checkout"); Instant logged = store.get("checkout"); assertTrue(Instant.now().isBefore(logged.plusSeconds(1))); // racy } ``` Fix. Inject a [`java.time.Clock`](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/time/Clock.html) into the production class so tests can swap in `Clock.fixed(...)`. The production wiring uses `Clock.systemUTC()`; the tests use a frozen clock and assert exactly. ``` class AuditLogger { private final Clock clock; AuditLogger(Clock clock) { this.clock = clock; } void log(String event) { store.put(event, clock.instant()); } } @Test void auditTimestampIsRecent() { Clock fixed = Clock.fixed(Instant.parse("2026-01-01T00:00:00Z"), ZoneOffset.UTC); AuditLogger logger = new AuditLogger(fixed); logger.log("checkout"); assertEquals(fixed.instant(), store.get("checkout")); } ``` With Mergify. Test Engine links timing failures to their CI runner type. When a test only fails on the slower runner pool and the failure is always within a small tolerance window, the dashboard surfaces the real-clock dependency. Pattern 7 ### MockWebServer instances that never get shut down Symptom. A long suite run eventually fails with \`BindException: Address already in use\` or \`Too many open files\`, and a rerun of the failing class passes. Root cause. [MockWebServer](https://github.com/square/okhttp/tree/master/mockwebserver) binds an ephemeral port and starts a background dispatcher thread. Forgetting to call `server.shutdown()` in `@AfterEach` leaks both. After enough tests the JVM hits the file-descriptor limit and the next test fails on socket creation. ``` class HttpClientTest { @Test void postsJson() throws Exception { MockWebServer server = new MockWebServer(); server.start(); server.enqueue(new MockResponse().setBody("{}")); // no shutdown client.post(server.url("/").toString(), "{}"); } } ``` Fix. Lift the server into a field with `@BeforeEach` + `@AfterEach`, or register it as a JUnit 5 extension so cleanup is guaranteed even when the test fails. ``` class HttpClientTest { private MockWebServer server; @BeforeEach void setUp() throws IOException { server = new MockWebServer(); server.start(); } @AfterEach void tearDown() throws IOException { server.shutdown(); } @Test void postsJson() throws Exception { server.enqueue(new MockResponse().setBody("{}")); client.post(server.url("/").toString(), "{}"); } } ``` With Mergify. Test Engine groups failures whose only signature is \`BindException\` or \`Too many open files\` into an fd-exhaustion bucket and surfaces the test that first triggered the leak rather than the random victim that ran when the JVM ran out. Pattern 8 ### Surefire rerunFailingTestsCount hiding real bugs Symptom. Your build is green. A user hits a bug your tests were supposed to catch. Root cause. [`surefire.rerunFailingTestsCount`](https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html) reruns failing tests up to N times and reports the last result. A real race that loses on attempt 1 and wins on attempt 2 gets reported as green. The bug is still there. The build has decided not to look at it. ``` maven-surefire-plugin 3 ``` Fix. Do not retry at the build level. When a test is genuinely flaky, fix it. When the fix takes longer than a session, quarantine it instead. That keeps the signal visible without blocking the merge queue. With Mergify. Test Engine reruns at the CI level with attempt-level result tracking. You see that a test passed on attempt 2 of 3, which is exactly the information \`rerunFailingTestsCount\` throws away. Quarantine kicks in once the pattern is clear. Detection ## How to detect flaky JUnit tests in CI Surefire and Gradle both emit JUnit XML out of the box. Point Mergify at the XML output of every CI run and Test Engine builds a confidence score for every test on your default branch. PR runs are compared against that baseline. Anything inconsistent gets flagged in a PR comment before the author merges. mergify ci ``` # 1. Surefire/Gradle already emit JUnit XML by default. Locate the file: # Maven: target/surefire-reports/TEST-*.xml # Gradle: build/test-results/test/TEST-*.xml # 2. Upload the result (once, in CI) curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh mergify ci junit-process target/surefire-reports/TEST-*.xml ``` Prevention ## How to prevent flaky JUnit tests from merging On every PR, Mergify reruns the tests whose confidence is below threshold, without Surefire's \`rerunFailingTestsCount\` touching your config. The PR gets a comment naming the unreliable tests, their confidence history, and whether the failure on this PR is new or historical noise. Authors fix the real bugs before merge instead of re-running CI until it passes. Quarantine ## How to quarantine flaky JUnit tests without skipping them Once a JUnit test is confirmed flaky, Test Engine quarantines it. The test still runs in the suite, no \`@Disabled\` rewrite required, but its result no longer blocks merges or marks the pipeline red. When the pass rate on main recovers, quarantine lifts automatically and the test goes back to being load-bearing. Your CI ## Run it on your CI The JUnit setup for Test Engine on each CI, how its retries compare with the CI's own reruns, and what makes JUnit flake on that CI's runners. - [JUnit on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions/junit) ## Want to see which JUnit tests in your repo are already flaky? Works with Surefire, Gradle, or any tool that emits JUnit XML. No extra plugins required. Setup takes under five minutes. [Book a discovery call](https://mergify.com/discovery) ## Frequently asked questions Why are my JUnit tests flaky in CI but pass locally? Your laptop and your CI runner differ in CPU count, parallelism, JDK build, and the order JUnit happens to pick for test methods. Tests that race on static state, hold references to a Spring context that another test evicts, or read the system clock surface those issues only under CI's tighter resource budget. Reproduce locally with \`mvn test -T 1C -Dsurefire.rerunFailingTestsCount=0\` (or the same Surefire config CI uses) to expose the failure, then fix the underlying coupling before pushing. How do I detect flaky JUnit tests? JUnit alone cannot tell flaky from broken since each run gives one data point per test. You need to run the same commit multiple times and compare results. Mergify Test Engine does that on every PR and on the default branch, scores each test, and surfaces the tests whose pass rate drops below a confidence threshold. Does Surefire's \`rerunFailingTestsCount\` fix flaky tests? No, it hides them. A test that fails on attempt 1 and passes on attempt 2 is still broken; you have only decided not to look at the failure. Use Surefire's \`rerunFailingTestsCount\` as a temporary bandage for a test you are actively fixing, never as a permanent policy. For visibility without blocking the merge queue, quarantine instead of retry. What causes flaky tests in JUnit? Eight patterns cover most of what we see: method-order assumptions without @TestMethodOrder, static-field state leaking across tests, Surefire forkCount and parallel resources races, Mockito mock leakage under @TestInstance(PER\_CLASS), @SpringBootTest context-cache surprises, time-based assertions without an injected Clock, MockWebServer instances that never get shut down, and Surefire rerunFailingTestsCount hiding real bugs. Each is covered above with a minimal reproducer. How do I quarantine a flaky JUnit test without deleting it? Mergify Test Engine quarantines the test automatically once its confidence score drops. The test still runs in the suite, but a failing result no longer blocks merges and its noise no longer drowns out real signal. When the test stabilizes on main, quarantine lifts automatically. No \`@Disabled\`, no commented-out tests, no orphaned files. How do I make JUnit 5 run tests in a deterministic order? Annotate the class with \`@TestMethodOrder(MethodOrderer.OrderAnnotation.class)\` and add \`@Order(N)\` to each method. But ordering is almost always a smell: a test class that needs ordering is a chain of dependent tests pretending to be independent. Refactor the prerequisite into \`@BeforeEach\` so each test stands alone, and reserve \`@TestMethodOrder\` for the rare case (a smoke test that walks an end-to-end happy path) where ordering is the assertion. ## Ship your JUnit suite green. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/minitest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/Minitest # Flaky tests in Minitest. Named, fixed, and quarantined. Flaky Minitest suites are not random. They follow patterns: parallel worker races, order-dependent escape hatches, transactional fixtures with JS specs, Mocha stubs that leak, travel\_to without travel\_back. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Published April 2026 Example PR comment from the Mergify bot detecting a flaky Minitest test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky Minitest suite: parallelize\_workers races on shared state, i\_suck\_and\_my\_tests\_are\_order\_dependent! masking real coupling, fixtures plus transactional tests in JS-driven specs, ActiveSupport::TestCase singleton leakage, Time.current without travel\_back, Mocha mocks that forgot to verify, setup/teardown skipped on early errors, and shell-level minitest reruns hiding real bugs. Each one has a minimal reproducer, a fix in stock Minitest, and a Mergify Test Engine detection strategy below. ## Why are Minitest tests flaky? [Minitest](https://github.com/minitest/minitest) is the default for Rails. It is small, fast, and shipped with Ruby itself. Tests are plain Ruby methods on an `ActiveSupport::TestCase` subclass, parallelism is opt-in via `parallelize(workers: :number_of_processors)`, and the runner shuffles tests by default. The minimalism is the appeal. It is also why most flakes come from Ruby's process-wide globals leaking between tests on the same worker. Layer on the Rails ecosystem. [Transactional tests](https://guides.rubyonrails.org/testing.html#transactional-tests) wrap each test in a database transaction and roll back at the end, but the rollback only works for tests that share the test connection: a JS-driven Capybara spec uses a separate connection and sees a half-empty database. [TimeHelpers](https://api.rubyonrails.org/classes/ActiveSupport/Testing/TimeHelpers.html), fixtures, and the Rails autoloader all carry their own flake categories that are easy to miss until ordering exposes them. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of Minitest suites: parallelize\_workers races on shared state, i\_suck\_and\_my\_tests\_are\_order\_dependent! masking real coupling, fixtures plus transactional tests in JS-driven specs, ActiveSupport::TestCase singleton leakage, Time.current without travel\_back, Mocha mocks that forgot to verify, setup/teardown skipped on early errors, and shell-level minitest reruns hiding real bugs. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01parallelize\_workers races on shared state](#pattern-parallelize-workers-races-on-shared-state) 2. [02i\_suck\_and\_my\_tests\_are\_order\_dependent! masking real coupling](#pattern-i-suck-and-my-tests-are-order-dependent-masking-real-coupling) 3. [03Fixtures plus transactional tests in JS-driven specs](#pattern-fixtures-plus-transactional-tests-in-js-driven-specs) 4. [04ActiveSupport::TestCase singleton leakage](#pattern-activesupport-testcase-singleton-leakage) 5. [05Time.current without travel\_back](#pattern-time-current-without-travel-back) 6. [06Mocha mocks that forgot to verify](#pattern-mocha-mocks-that-forgot-to-verify) 7. [07setup/teardown skipped on early errors](#pattern-setup-teardown-skipped-on-early-errors) 8. [08Shell-level minitest reruns hiding real bugs](#pattern-shell-level-minitest-reruns-hiding-real-bugs) Pattern 1 ### parallelize\_workers races on shared state Symptom. A test that mutates a class-level cache passes alone and fails after \`parallelize(workers: :number\_of\_processors)\` is enabled, with assertions seeing values from a sibling test. Root cause. [parallelize](https://api.rubyonrails.org/classes/ActiveSupport/Testing/Parallelization.html) forks (or threads) the test runner. Anything those workers share inside Ruby (a class variable, a thread-local that leaks across the fiber boundary, a singleton instance) becomes a race. Forks are isolated, so most state is per-worker, but tests that hit the same external resource (a fixed Redis key, a shared file) collide just like they would in any parallel test framework. ``` class CounterTest < ActiveSupport::TestCase parallelize(workers: 4) @@count = 0 test "increments the counter" do @@count += 1 assert_equal 1, @@count end test "increments again" do @@count += 1 assert_equal 1, @@count # fails when sibling already incremented end end ``` Fix. Move shared state into per-test instance variables (set in `setup`) or scope external resources by `name`. For genuine global side effects, use `parallelize_setup` + `parallelize_teardown` hooks to give each worker its own database, Redis namespace, or temp directory. ``` class CounterTest < ActiveSupport::TestCase parallelize(workers: 4) setup { @count = 0 } test "increments the counter" do @count += 1 assert_equal 1, @count end end ``` With Mergify. Test Engine tags failures that only appear under parallel runs as parallelism-sensitive and never under sequential reruns. The dashboard surfaces the parallel-only signature so you know to look for class-level state. Pattern 2 ### i\_suck\_and\_my\_tests\_are\_order\_dependent! masking real coupling Symptom. A legacy test class is annotated with the order-dependent escape hatch and passes; removing the hatch reveals tests that depend on each other in non-obvious ways. Root cause. Minitest randomizes test order to expose hidden coupling. [`i_suck_and_my_tests_are_order_dependent!`](https://github.com/minitest/minitest/blob/master/lib/minitest/test.rb) turns the shuffle off for a class. The name is intentionally hostile because the only honest reason to use it is that the suite has a coupling bug you are not ready to fix. Leaving it in place hides the bug; the next refactor changes the implicit order and the test fails for what looks like an unrelated reason. ``` class LegacyOrderTest < ActiveSupport::TestCase i_suck_and_my_tests_are_order_dependent! test "1: creates a user" do @@user = User.create!(name: "Rémy") end test "2: reads the user" do # depends on test 1 having run first; class variable smuggles state assert_equal "Rémy", @@user.name end end ``` Fix. Refactor to make every test independent: build state in `setup` or inside the test, drop the order-dependent annotation, and let the shuffle catch any remaining coupling. ``` class LegacyOrderTest < ActiveSupport::TestCase setup do @user = User.create!(name: "Rémy") end test "creates a user" do assert @user.persisted? end test "reads the user" do assert_equal "Rémy", @user.name end end ``` With Mergify. Test Engine records the seed for every run. When a class still depends on order after the annotation is removed, failures cluster around specific seed ranges; the dashboard surfaces them so the missed coupling is easy to find. Pattern 3 ### Fixtures plus transactional tests in JS-driven specs Symptom. A system test annotated with \`js: true\` (Capybara + Selenium/Cuprite) sees a half-empty database; the data the test created in a fixture or \`setup\` block is missing in the browser. Root cause. Rails wraps every test in a database transaction (`use_transactional_tests = true`) and rolls back at the end. The transaction holds on the test process's connection, so any test that runs Rails code inside the test process sees the data. A Capybara JS driver opens a separate browser connection that cannot see uncommitted data from another connection. The browser logs in, the user does not exist on its connection, the test fails. ``` class CheckoutSystemTest < ApplicationSystemTestCase driven_by :selenium, using: :headless_chrome setup do @user = User.create!(email: "user@example.com", password: "secret") end test "user signs in" do visit "/login" fill_in "Email", with: @user.email # browser cannot see @user; sign-in fails with 'invalid credentials' fill_in "Password", with: "secret" click_button "Sign in" assert_text "Welcome back" end end ``` Fix. Disable transactional tests for system specs and use a truncation-based cleaner, or use Rails 7.1+'s [connection-shared transactions](https://api.rubyonrails.org/classes/ActiveRecord/TestFixtures/ClassMethods.html#method-i-use_transactional_tests). The simplest path: switch the cleaning strategy via `database_cleaner-active_record`'s `truncation` for JS specs. ``` # test/test_helper.rb class ActionDispatch::SystemTestCase self.use_transactional_tests = false setup { DatabaseCleaner.strategy = :truncation; DatabaseCleaner.start } teardown { DatabaseCleaner.clean } end ``` With Mergify. Test Engine catches the signature: a system test fails consistently with empty-database errors and passes when re-run with the rest of the suite cleared (truncation from the previous run masks the issue). The dashboard surfaces the affected test type by tag. Pattern 4 ### ActiveSupport::TestCase singleton leakage Symptom. A test that stubs \`Rails.application.config.feature\_x\` passes alone and breaks an unrelated test in another file with a value the second test never set. Root cause. Rails configuration objects are global singletons. A test that mutates `Rails.application.config.x` or stubs a method on a class without unstubbing leaves that change in place for every later test in the worker. Class reopening, monkey patching, and constant reassignment all live for the life of the process. ``` class FeatureFlagTest < ActiveSupport::TestCase test "with the new path" do Rails.application.config.use_new_billing = true assert_equal 49, Pricing.for(:pro) end # no teardown; config stays set end class LegacyBillingTest < ActiveSupport::TestCase test "with the legacy path" do # expects use_new_billing == false (default). actual: true, leaked. assert_equal 99, Pricing.for(:pro) end end ``` Fix. Use a temporary stub helper that auto-restores: `stub_const` from `activesupport`'s testing helpers, or a per-test reset in `teardown`. For configuration, capture the original value and restore it explicitly. ``` class FeatureFlagTest < ActiveSupport::TestCase setup do @original = Rails.application.config.use_new_billing Rails.application.config.use_new_billing = true end teardown do Rails.application.config.use_new_billing = @original end test "with the new path" do assert_equal 49, Pricing.for(:pro) end end ``` With Mergify. Test Engine groups the downstream failures by the test they all follow. When five seemingly unrelated tests fail only after a config-mutating test runs first, the dashboard surfaces the upstream culprit instead of pointing at five red herrings. Pattern 5 ### Time.current without travel\_back Symptom. A test that uses \`travel\_to\` passes, and the next test that touches \`Time.current\` fails with a date months in the past or future. Root cause. [`travel_to`](https://api.rubyonrails.org/classes/ActiveSupport/Testing/TimeHelpers.html#method-i-travel_to) moves the clock for every later `Time.current` call in the worker. Calling `travel_to` inside a test without a paired `travel_back` (or without using the block form) leaves the clock frozen for every test that runs after on the same worker. ``` class InvitationTest < ActiveSupport::TestCase test "expires after 7 days" do travel_to Time.zone.parse("2026-01-01") invite = Invitation.create! travel 8.days assert invite.expired? # missing travel_back end end class SessionTest < ActiveSupport::TestCase test "session token is valid for an hour" do token = SessionToken.new(@user) # Time.current is still January 9 2026; assertion fails assert_in_delta 1.hour.from_now, token.expires_at, 1.minute end end ``` Fix. Use the block form of `travel_to` so the clock auto-restores at the end of the block. Or include a global `teardown { travel_back }` in the base test case. ``` # test/test_helper.rb class ActiveSupport::TestCase teardown { travel_back } # belt-and-braces end # in the test test "expires after 7 days" do travel_to Time.zone.parse("2026-01-01") do invite = Invitation.create! travel 8.days assert invite.expired? end end ``` With Mergify. Test Engine shows the cross-test time signature: a test fails only when run after a known time-mutating test, and only when its assertions touch the clock. The dashboard surfaces the ordering so the missed travel\_back is easy to locate. Pattern 6 ### Mocha mocks that forgot to verify Symptom. A test asserts a mock was called, the assertion passes, but a sibling test fails on a stubbed method that was never declared in its own scope. Root cause. [Mocha](https://github.com/freerange/mocha) stubs methods globally on the receiver instance or class. Without `mocha/minitest`'s automatic teardown wiring, a stub set in test A persists into test B. Even with the wiring, calling `stubs(:method)` on a class object outside an `Object#expects` block leaks if the test exits early. ``` class BillingTest < ActiveSupport::TestCase test "charges the user" do StripeClient.stubs(:charge).returns(true) assert Billing.charge_user(42) # exits without unstub; StripeClient.charge stays mocked end test "falls back when stripe is down" do # StripeClient.charge is still the stub returning true assert Billing.fallback_for(42) end end ``` Fix. Require `mocha/minitest` in your test helper so stubs auto-revert. Prefer `expects` over `stubs` when you actually want to verify the call: failed expectations fail the test loudly. ``` # test/test_helper.rb require "mocha/minitest" # in the test test "charges the user" do StripeClient.expects(:charge).with(42).returns(true) assert Billing.charge_user(42) # mocha verifies and unstubs at end of test end ``` With Mergify. Test Engine catches the cross-test signature: a test only fails when run after a specific other test, and only when its assertion involves a stubbed method. The dashboard tags the dependency so you know it's mock leakage, not a real regression. Pattern 7 ### setup/teardown skipped on early errors Symptom. A test class that opens a database connection in \`setup\` leaves orphan connections after a \`setup\` block raises, and the next CI run starts with a connection-pool warning. Root cause. Minitest's `teardown` runs even when the test body fails, but it does not run when `setup` itself raises (Minitest never reaches teardown if setup blew up). External resources opened in setup before the failing line stay open. `Minitest::Test#after_teardown` has the same constraint. ``` class APIClientTest < ActiveSupport::TestCase setup do @conn = open_connection_pool @client = APIClient.new(maybe_nil: nil) # raises NoMethodError end teardown do @conn.close # never runs; @conn leaks end end ``` Fix. Wrap risky construction in Ruby's `begin/ensure` so the connection is always closed before the test exits, even when setup raises. The classic ActiveSupport `around`\-style helper does the same thing implicitly with `yield`. ``` class APIClientTest < ActiveSupport::TestCase setup do @conn = open_connection_pool begin @client = APIClient.new(maybe_nil: nil) rescue @conn.close # ensure the resource is freed even when init raises raise end end teardown do @conn.close if @conn && !@conn.closed? end end ``` With Mergify. Test Engine groups failures whose only signature is connection-pool exhaustion or 'too many open files' into an fd-leak bucket and surfaces the test that first triggered the leak, rather than the random victim that ran when the JVM ran out. Pattern 8 ### Shell-level minitest reruns hiding real bugs Symptom. Your CI runs \`bundle exec rails test || bundle exec rails test\` and the build is green. A user hits a bug your tests should have caught. Root cause. Minitest does not ship with a retry mechanism, so teams reach for shell-level retries: rerun the failing test with the same seed, or rerun the whole suite if anything failed. A real race that loses on attempt 1 and wins on attempt 2 gets reported as green. The bug is still there. The pipeline has decided not to look at it. ``` # CI script (please don't) bundle exec rails test || bundle exec rails test || exit 1 ``` Fix. Do not retry at the shell level. When a test is genuinely flaky, fix it. When the fix takes longer than a session, quarantine it instead. That keeps the signal visible without blocking the merge queue. With Mergify. Test Engine reruns at the CI level with attempt-level result tracking. You see that a test passed on attempt 2 of 3, which is exactly the information shell-level retries throw away. Quarantine kicks in once the pattern is clear. Detection ## How to detect flaky Minitest tests in CI Add the minitest-reporters gem and configure a JUnit reporter, then upload the resulting XML to Mergify with a one-line CLI call. Test Engine builds a confidence score for every test on your default branch. PR runs are compared against that baseline. Anything inconsistent gets flagged in a PR comment before the author merges. mergify ci ``` # 1. Add the JUnit reporter to your Gemfile gem "minitest-reporters", group: :test # 2. Wire it up in test/test_helper.rb require "minitest/reporters" Minitest::Reporters.use! [Minitest::Reporters::JUnitReporter.new] # 3. Upload the result (once, in CI) curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh mergify ci junit-process test/reports/*.xml ``` Prevention ## How to prevent flaky Minitest tests from merging On every PR, Mergify reruns the tests whose confidence is below threshold, without shell-level minitest retries touching your config. The PR gets a comment naming the unreliable tests, their confidence history, and whether the failure on this PR is new or historical noise. Authors fix the real bugs before merge instead of re-running CI until it passes. Quarantine ## How to quarantine flaky Minitest tests without skipping them Once a Minitest test is confirmed flaky, Test Engine quarantines it. The test still runs in the suite, no \`skip\` rewrite required, but its result no longer blocks merges or marks the pipeline red. When the pass rate on main recovers, quarantine lifts automatically and the test goes back to being load-bearing. Your CI ## Run it on your CI The Minitest setup for Test Engine on each CI, how its retries compare with the CI's own reruns, and what makes Minitest flake on that CI's runners. - [Minitest on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions/minitest) ## Want to see which Minitest tests in your repo are already flaky? Works with minitest-reporters or any JUnit-compatible Minitest reporter. Setup takes under five minutes. [Book a discovery call](https://mergify.com/discovery) ## Frequently asked questions Why are my Minitest tests flaky in CI but pass locally? Your laptop and your CI runner differ in CPU count, parallel worker count, and the random seed Minitest picked for the run. Tests that mutate global Rails configuration, leave Mocha stubs unfreed, or depend on transactional fixtures from a different database connection surface those issues only under CI's tighter resource budget. Reproduce locally with the failing seed (\`bundle exec rails test --seed=N\`) and bisect until the minimal pair shows up. How do I detect flaky Minitest tests? Minitest alone cannot tell flaky from broken since each run gives one data point per test. You need to run the same commit multiple times and compare results. Mergify Test Engine does that on every PR and on the default branch, scores each test, and surfaces the tests whose pass rate drops below a confidence threshold. Does shell-level minitest retries fix flaky tests? No, it hides them. A test that fails on attempt 1 and passes on attempt 2 is still broken; you have only decided not to look at the failure. Use shell-level minitest retries as a temporary bandage for a test you are actively fixing, never as a permanent policy. For visibility without blocking the merge queue, quarantine instead of retry. What causes flaky tests in Minitest? Eight patterns cover most of what we see: parallelize\_workers races on shared state, i\_suck\_and\_my\_tests\_are\_order\_dependent! masking real coupling, fixtures plus transactional tests in JS-driven specs, ActiveSupport::TestCase singleton leakage, Time.current without travel\_back, Mocha mocks that forgot to verify, setup/teardown skipped on early errors, and shell-level minitest reruns hiding real bugs. Each is covered above with a minimal reproducer. How do I quarantine a flaky Minitest test without deleting it? Mergify Test Engine quarantines the test automatically once its confidence score drops. The test still runs in the suite, but a failing result no longer blocks merges and its noise no longer drowns out real signal. When the test stabilizes on main, quarantine lifts automatically. No \`skip\`, no commented-out tests, no orphaned files. How do I run Rails system tests in parallel without database conflicts? Each parallel worker needs its own database. Rails sets up \`parallelize\_setup\` to create per-worker databases automatically when you call \`parallelize(workers: :number\_of\_processors)\`, but only when \`use\_transactional\_tests\` works for your specs. For Capybara JS-driven specs, switch the cleaning strategy to truncation (or deletion) per worker so the browser's separate connection sees committed data. ## Ship your Minitest suite green. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/mstest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/MSTest # Flaky tests in MSTest. Named, fixed, and quarantined. Flaky MSTest suites are not random. They follow patterns: AssemblyInitialize coupling to discovery order, ClassInitialize racing on static state, DeploymentItem path collisions, async void test signatures, real-clock assertions. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Published April 2026 Example PR comment from the Mergify bot detecting a flaky MSTest test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky MSTest suite: AssemblyInitialize racing test discovery, ClassInitialize ordering surprises across parallel classes, DeploymentItem races on shared files, TestContext mutated across tests, parallel execution settings interacting with shared state, async test signatures missing the Task return, DateTime.UtcNow without TimeProvider, and TestProperty rerun configuration hiding real bugs. Each one has a minimal reproducer, a fix in stock MSTest, and a Mergify Test Engine detection strategy below. ## Why are MSTest tests flaky? [MSTest](https://learn.microsoft.com/dotnet/core/testing/unit-testing-with-mstest) is Microsoft's first-party .NET test framework. It ships with Visual Studio and the .NET SDK, so most enterprise .NET teams use it by default. The default model runs every test against a fresh instance of the test class but keeps the AppDomain alive across tests, which means static state, the system clock, and any singleton survive the entire run. Two facets cause most flakes. [`[AssemblyInitialize]`](https://learn.microsoft.com/dotnet/api/microsoft.visualstudio.testtools.unittesting.assemblyinitializeattribute) runs once per test assembly with no guarantees about which test it precedes; if it depends on test discovery order, an unrelated change to test selection breaks it. And [parallelization](https://learn.microsoft.com/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file) is configured in `.runsettings` with subtle interactions: MSTest still creates a fresh test class instance per test method, but parallel execution exposes races through the assembly's static state, singletons, shared files or databases, and other process-wide configuration. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of MSTest suites: AssemblyInitialize racing test discovery, ClassInitialize ordering surprises across parallel classes, DeploymentItem races on shared files, TestContext mutated across tests, parallel execution settings interacting with shared state, async test signatures missing the Task return, DateTime.UtcNow without TimeProvider, and TestProperty rerun configuration hiding real bugs. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01AssemblyInitialize racing test discovery](#pattern-assemblyinitialize-racing-test-discovery) 2. [02ClassInitialize ordering surprises across parallel classes](#pattern-classinitialize-ordering-surprises-across-parallel-classes) 3. [03DeploymentItem races on shared files](#pattern-deploymentitem-races-on-shared-files) 4. [04TestContext mutated across tests](#pattern-testcontext-mutated-across-tests) 5. [05Parallel execution settings interacting with shared state](#pattern-parallel-execution-settings-interacting-with-shared-state) 6. [06Async test signatures missing the Task return](#pattern-async-test-signatures-missing-the-task-return) 7. [07DateTime.UtcNow without TimeProvider](#pattern-datetime-utcnow-without-timeprovider) 8. [08TestProperty rerun configuration hiding real bugs](#pattern-testproperty-rerun-configuration-hiding-real-bugs) Pattern 1 ### AssemblyInitialize racing test discovery Symptom. A test that depends on a database container started in \`\[AssemblyInitialize\]\` fails on the first run after the test runner discovered tests in a different order. Root cause. [`[AssemblyInitialize]`](https://learn.microsoft.com/dotnet/api/microsoft.visualstudio.testtools.unittesting.assemblyinitializeattribute) fires once per assembly, before any test in that assembly runs. It receives a `TestContext` tied to assembly initialization, not to a specific test, so any code that pulls per-test data from it (deployment items, test categories, properties from a particular test method) reads the wrong values or none at all. The hazard is the global initialization itself: ordering relative to `[ClassInitialize]`, side effects on shared resources before tests start, and brittle assumptions about what the runner has set up by then. ``` [TestClass] public class GlobalSetup { [AssemblyInitialize] public static async Task Init(TestContext ctx) { // assumes the first test brings the right deployment items; // breaks when discovery picks a different test first var data = ctx.DeploymentDirectory + "/seed.json"; await Database.SeedAsync(File.ReadAllText(data)); } } ``` Fix. Make `[AssemblyInitialize]` independent of which test runs first. Hardcode resource paths or load fixtures from a known absolute location. For per-test deployment items, use `[DeploymentItem]` on the test method itself and read inside `[TestInitialize]`. ``` [TestClass] public class GlobalSetup { [AssemblyInitialize] public static async Task Init(TestContext ctx) { await Database.SeedAsync(EmbeddedResources.Read("seed.json")); } } [TestClass] public class CheckoutTests { [DeploymentItem("Resources/checkout-fixtures.json")] [TestMethod] public async Task PlacesOrder() { ... } } ``` With Mergify. Test Engine tags failures whose only signature is \`AssemblyInitialize threw an exception\` and groups them by the file the failing test sits in. The dashboard surfaces the discovery-order coupling so the brittle assumption is the obvious lead. Pattern 2 ### ClassInitialize ordering surprises across parallel classes Symptom. Tests that depend on \`\[ClassInitialize\]\` setup pass alone and fail under \`\` with assertions about state another class initialized. Root cause. [`[ClassInitialize]`](https://learn.microsoft.com/dotnet/api/microsoft.visualstudio.testtools.unittesting.classinitializeattribute) fires once per class. With `Scope="ClassLevel"` in `.runsettings`, two classes can run their `[ClassInitialize]` hooks concurrently. If they both touch a shared static (a connection pool, a config dictionary, the Mock framework's global registry), they race. ``` [TestClass] public class CheckoutTests { [ClassInitialize] public static void Init(TestContext _) { Bootstrap.Configure(env: "checkout"); // mutates a static AppConfig } } [TestClass] public class InventoryTests { [ClassInitialize] public static void Init(TestContext _) { Bootstrap.Configure(env: "inventory"); // races with CheckoutTests } } // runsettings: ClassLevel ``` Fix. Avoid mutating shared static state in `[ClassInitialize]`. Refactor configuration into instance-per-test scope, or wrap the mutating section in `lock` with explicit ordering. For test classes that genuinely conflict, mark them `[DoNotParallelize]`. ``` [TestClass] [DoNotParallelize] public class CheckoutTests { [ClassInitialize] public static void Init(TestContext _) { Bootstrap.Configure(env: "checkout"); } } ``` With Mergify. Test Engine tags failures that only appear with parallel runsettings as parallelism-sensitive. The dashboard surfaces the parallel-only signature so the \[DoNotParallelize\] decision is straightforward. Pattern 3 ### DeploymentItem races on shared files Symptom. A test that copies a fixture file via \`\[DeploymentItem\]\` passes alone and fails when run alongside another test that writes to the same path. Root cause. [`[DeploymentItem]`](https://learn.microsoft.com/dotnet/api/microsoft.visualstudio.testtools.unittesting.deploymentitemattribute) copies a file or directory into `TestContext.DeploymentDirectory` before the test runs. The deployment directory is shared across tests in the same run by default. If two parallel tests both write to `DeploymentDirectory/output.json`, they race; the loser sees the wrong content or a partially written file. ``` [TestClass] public class ExporterTests { [TestMethod] [DeploymentItem("Fixtures/template.json")] public void ExportsOrder() { var output = Path.Combine(TestContext.DeploymentDirectory, "output.json"); File.WriteAllText(output, exporter.Render(order)); Assert.IsTrue(File.Exists(output)); } [TestMethod] [DeploymentItem("Fixtures/template.json")] public void ExportsInvoice() { var output = Path.Combine(TestContext.DeploymentDirectory, "output.json"); File.WriteAllText(output, exporter.Render(invoice)); // sometimes asserts on content from ExportsOrder under parallel StringAssert.Contains(File.ReadAllText(output), "INV-"); } } ``` Fix. Write to a per-test path derived from `TestContext.TestName` or `Path.GetTempFileName()`. Reserve `DeploymentDirectory` for read-only fixtures. ``` [TestMethod] public void ExportsOrder() { var output = Path.Combine(Path.GetTempPath(), $"{TestContext.TestName}.json"); File.WriteAllText(output, exporter.Render(order)); Assert.IsTrue(File.Exists(output)); } ``` With Mergify. Test Engine groups failures whose only signature is wrong file content or \`IOException: file in use\` into a per-suite bucket. The dashboard surfaces the test that first wrote to the shared path so the per-test path fix lands at the source. Pattern 4 ### TestContext mutated across tests Symptom. A test that reads \`TestContext.Properties\["tenantId"\]\` passes alone and fails inside the suite with a tenant set by a sibling. Root cause. [`TestContext.Properties`](https://learn.microsoft.com/dotnet/api/microsoft.visualstudio.testtools.unittesting.testcontext) is a per-test bag, but tests that mutate it in `[TestInitialize]` via reflection or shared helpers can leak the mutation across tests when those helpers cache the context. Combined with parallel runs, two tests on the same thread see each other's properties. ``` [TestClass] public class TenantTests { public TestContext TestContext { get; set; } [TestInitialize] public void Setup() { TestContext.Properties["tenantId"] = TenantHelper.PickRandom().Id; } [TestMethod] public void ProcessesOrder() { var id = (string)TestContext.Properties["tenantId"]; // sometimes the wrong tenant under parallel runs Assert.IsNotNull(orderService.For(id).Process()); } } ``` Fix. Pass per-test values through instance fields, not `TestContext.Properties`. `TestContext` is meant for run metadata (output, deployment paths), not for state your tests need to read deterministically. ``` [TestClass] public class TenantTests { private string _tenantId; [TestInitialize] public void Setup() { _tenantId = TenantHelper.PickRandom().Id; } [TestMethod] public void ProcessesOrder() { Assert.IsNotNull(orderService.For(_tenantId).Process()); } } ``` With Mergify. Test Engine catches the cross-test signature: a test only fails after a specific other test has run, with assertions about a value the failing test never set. The dashboard surfaces the predecessor so the misused TestContext is the obvious lead. Pattern 5 ### Parallel execution settings interacting with shared state Symptom. A test suite with \`4\` in \`.runsettings\` fails with assertions about a static counter another test incremented. Root cause. MSTest v2's parallel scope (`MethodLevel` or `ClassLevel`) and worker count come from [.runsettings](https://learn.microsoft.com/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file). With `MethodLevel`, test methods can run concurrently, but MSTest still creates a separate test class instance per test method. The race is in shared process-wide state: static fields, registered services, the Mock framework's verifier, and any external resource the methods touch. ``` // .runsettings 4 MethodLevel [TestClass] public class CounterTests { private static int _counter; [TestInitialize] public void Reset() { _counter = 0; } [TestMethod] public void IncrementsToOne() { _counter++; Assert.AreEqual(1, _counter); // sometimes 2 under parallel methods } } ``` Fix. Pick parallelism that matches the suite's isolation. Method-level requires every test to be a pure function of its inputs. For class-level (one thread per class), keep the static state but ensure each class owns it. For per-test isolation, prefer instance fields and local variables. ``` // .runsettings 4 ClassLevel ``` With Mergify. Test Engine tags failures that only appear with method-level parallelism and pass under class-level reruns as parallelism-sensitive. The dashboard surfaces the parallel-only signature so shared static or other process-wide mutable state is the obvious lead. Pattern 6 ### Async test signatures missing the Task return Symptom. An async test marked \`\[TestMethod\] public async void\` passes locally and reports false positives in CI (a test that should have failed reports as PASSED). Root cause. MSTest awaits async test methods only when they return `Task` or `ValueTask`. An `async void` test method returns immediately; the test runner marks it PASSED before the async body completes. Any assertion failure inside the body is thrown on the synchronization context, where MSTest never sees it. ``` [TestMethod] public async void ChargesUser() { var ok = await billingService.ChargeAsync(42); Assert.IsTrue(ok); // never observed; method returns void immediately } ``` Fix. Always return `Task` from async test methods. Treat `async void` in test code as a bug; some MSTest versions warn but most pass it through. ``` [TestMethod] public async Task ChargesUser() { var ok = await billingService.ChargeAsync(42); Assert.IsTrue(ok); } ``` With Mergify. Test Engine catches false positives by comparing the test's reported duration to assertions logged. A test that reports PASSED with a sub-millisecond duration despite calling async code is the fingerprint, and the dashboard flags it as suspect-pass. Pattern 7 ### DateTime.UtcNow without TimeProvider Symptom. A test that asserts an event happened "within the last second" passes locally and fails on the slower CI runner with a timestamp 1.2 seconds old. Root cause. Calling `DateTime.UtcNow` directly inside production code reads the system clock. A test that asserts on a freshly created timestamp races against scheduler jitter. Locally the assertion fires within a millisecond; on CI the same code runs after a longer pause and the assertion's tolerance window is too tight. ``` public class AuditLogger { public void Log(string evt) => store[evt] = DateTime.UtcNow; } [TestMethod] public void AuditTimestampIsRecent() { auditLogger.Log("checkout"); var logged = store["checkout"]; Assert.IsTrue((DateTime.UtcNow - logged).TotalMilliseconds < 50); // racy under CI scheduler jitter } ``` Fix. Inject [`TimeProvider`](https://learn.microsoft.com/dotnet/api/system.timeprovider) (built-in since .NET 8) into the production class so tests can swap in a fixed clock. Production wires `TimeProvider.System`; tests use `FakeTimeProvider` from the [Microsoft.Extensions.TimeProvider.Testing](https://learn.microsoft.com/dotnet/api/microsoft.extensions.time.testing.faketimeprovider) package, or a small custom subclass when adding the dependency is overkill. ``` public class AuditLogger { private readonly TimeProvider _clock; public AuditLogger(TimeProvider clock) { _clock = clock; } public void Log(string evt) => store[evt] = _clock.GetUtcNow(); } [TestMethod] public void AuditTimestamp() { var fake = new FakeTimeProvider(DateTimeOffset.Parse("2026-01-01Z")); var logger = new AuditLogger(fake); logger.Log("checkout"); Assert.AreEqual(fake.GetUtcNow(), store["checkout"]); } ``` With Mergify. Test Engine links timing failures to their CI runner type. When a test only fails on the slower runner pool and never on the laptop pool, the dashboard surfaces the resource sensitivity so the real-clock dependency is the obvious place to look. Pattern 8 ### TestProperty rerun configuration hiding real bugs Symptom. Your build is green. A user hits a bug that should have been caught by a test that ran four times yesterday before passing. Root cause. [`MaxCpuCount` + retry plugins](https://learn.microsoft.com/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file#mstest-element) for MSTest add a rerun-on-failure flag. A real race that loses on attempt 1 and wins on attempt 2 gets reported as PASSED. The bug is still there. The pipeline has decided not to look at it. ``` // .runsettings (please don't) 3600000 3 ``` Fix. Do not retry at the framework level. When a test is genuinely flaky, fix it. When the fix takes longer than a session, quarantine it instead. That keeps the signal visible without blocking the merge queue. With Mergify. Test Engine reruns at the CI level with attempt-level result tracking. You see that a test passed on attempt 2 of 3, which is exactly the information \`MSTestRetry\` configurations throw away. Quarantine kicks in once the pattern is clear. Detection ## How to detect flaky MSTest tests in CI MSTest's \`dotnet test\` runner emits TRX or JUnit XML reports depending on the logger. Configure the JUnit logger and upload the resulting XML to Mergify with a one-line CLI call. Test Engine builds a confidence score for every test on your default branch. PR runs are compared against that baseline. Anything inconsistent gets flagged in a PR comment before the author merges. mergify ci ``` # 1. Add the JUnit logger dotnet add package JUnitTestLogger # 2. Emit JUnit XML on every CI run dotnet test --logger "junit;LogFilePath=junit.xml" # 3. Upload the result (once, in CI) curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh mergify ci junit-process junit.xml ``` Prevention ## How to prevent flaky MSTest tests from merging On every PR, Mergify reruns the tests whose confidence is below threshold, without MSTest retry plugins touching your config. The PR gets a comment naming the unreliable tests, their confidence history, and whether the failure on this PR is new or historical noise. Authors fix the real bugs before merge instead of re-running CI until it passes. Quarantine ## How to quarantine flaky MSTest tests without skipping them Once a MSTest test is confirmed flaky, Test Engine quarantines it. The test still runs in the suite, no \`\[Ignore\]\` rewrite required, but its result no longer blocks merges or marks the pipeline red. When the pass rate on main recovers, quarantine lifts automatically and the test goes back to being load-bearing. Your CI ## Run it on your CI The MSTest setup for Test Engine on each CI, how its retries compare with the CI's own reruns, and what makes MSTest flake on that CI's runners. - [MSTest on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions/mstest) ## Want to see which MSTest tests in your repo are already flaky? Works with the JUnit logger or any JUnit-compatible MSTest reporter. Setup takes under five minutes. [Book a discovery call](https://mergify.com/discovery) ## Frequently asked questions Why are my MSTest tests flaky in CI but pass locally? Your laptop and your CI runner differ in CPU count, parallel scope settings, and which test the runner happened to discover first. MSTest tests that share static state across classes, race on DeploymentItem paths, or read DateTime.UtcNow surface those issues only under CI's tighter resource budget. Reproduce locally with \`dotnet test --settings ci.runsettings\` to apply the same parallel and discovery configuration as CI, then fix the underlying coupling before pushing. How do I detect flaky MSTest tests? MSTest alone cannot tell flaky from broken since each run gives one data point per test. You need to run the same commit multiple times and compare results. Mergify Test Engine does that on every PR and on the default branch, scores each test, and surfaces the tests whose pass rate drops below a confidence threshold. Does MSTest retry plugins fix flaky tests? No, it hides them. A test that fails on attempt 1 and passes on attempt 2 is still broken; you have only decided not to look at the failure. Use MSTest retry plugins as a temporary bandage for a test you are actively fixing, never as a permanent policy. For visibility without blocking the merge queue, quarantine instead of retry. What causes flaky tests in MSTest? Eight patterns cover most of what we see: AssemblyInitialize racing test discovery, ClassInitialize ordering surprises across parallel classes, DeploymentItem races on shared files, TestContext mutated across tests, parallel execution settings interacting with shared state, async test signatures missing the Task return, DateTime.UtcNow without TimeProvider, and TestProperty rerun configuration hiding real bugs. Each is covered above with a minimal reproducer. How do I quarantine a flaky MSTest test without deleting it? Mergify Test Engine quarantines the test automatically once its confidence score drops. The test still runs in the suite, but a failing result no longer blocks merges and its noise no longer drowns out real signal. When the test stabilizes on main, quarantine lifts automatically. No \`\[Ignore\]\`, no commented-out tests, no orphaned files. What's the difference between MSTest method-level and class-level parallelism? Method-level (\`MethodLevel\`) parallelizes individual test methods; class-level (\`ClassLevel\`) parallelizes test classes. MSTest still creates a separate test class instance per test method either way, so the main flakiness risk in either mode is shared static state or process-wide resources (files, environment variables, clocks, deployment paths). Method-level is often faster for pure tests; class-level can reduce contention when tests in the same class touch the same shared resources. Pick based on what your tests actually share, and mark conflicting classes \`\[DoNotParallelize\]\`. ## Ship your MSTest suite green. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/nunit [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/NUnit # Flaky tests in NUnit. Named, fixed, and quarantined. Flaky NUnit suites are not random. They follow patterns: parallel scope confusion, static field leakage, SetUpFixture cleanup that never fires, .Result deadlocks in async tests, real-clock assertions. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Published April 2026 Example PR comment from the Mergify bot detecting a flaky NUnit test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky NUnit suite: Parallelizable scope confusion across fixtures, static field state surviving the test run, SetUpFixture lifecycle surprises, TestContext attribute leakage between tests, async tests that .Result-deadlock on the sync context, DateTime.UtcNow without an injected IClock, HttpClient instances shared without per-test handlers, and Retry attribute hiding real bugs. Each one has a minimal reproducer, a fix in stock NUnit, and a Mergify Test Engine detection strategy below. ## Why are NUnit tests flaky? [NUnit](https://docs.nunit.org/) is the original .NET testing framework, predating MSTest and xUnit. It runs every test against a fresh instance of the test class by default, but the AppDomain (or AssemblyLoadContext on .NET Core+) lives for the entire run, which means every static field, singleton, and process-wide configuration survives across tests. NUnit's parallelism is opt-in via [`[Parallelizable]`](https://docs.nunit.org/articles/nunit/writing-tests/attributes/parallelizable.html), with three scopes (Self, Children, Fixtures) that compose subtly. Two facets cause most flakes. The parallelism scope rules look obvious until two fixtures with different scopes interact: a fixture marked `ParallelScope.Self` can run alongside a fixture marked `ParallelScope.None`, and shared static state turns the parallelism into a race. [async/await](https://learn.microsoft.com/dotnet/csharp/asynchronous-programming/) in test bodies is fine when the test is async, but mixing `.Result` or `.Wait()` on a Task that resumes on the captured sync context deadlocks under NUnit's runner. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of NUnit suites: Parallelizable scope confusion across fixtures, static field state surviving the test run, SetUpFixture lifecycle surprises, TestContext attribute leakage between tests, async tests that .Result-deadlock on the sync context, DateTime.UtcNow without an injected IClock, HttpClient instances shared without per-test handlers, and Retry attribute hiding real bugs. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01Parallelizable scope confusion across fixtures](#pattern-parallelizable-scope-confusion-across-fixtures) 2. [02Static field state surviving the test run](#pattern-static-field-state-surviving-the-test-run) 3. [03SetUpFixture lifecycle surprises](#pattern-setupfixture-lifecycle-surprises) 4. [04TestContext attribute leakage between tests](#pattern-testcontext-attribute-leakage-between-tests) 5. [05Async tests that .Result-deadlock on the sync context](#pattern-async-tests-that-result-deadlock-on-the-sync-context) 6. [06DateTime.UtcNow without an injected IClock](#pattern-datetime-utcnow-without-an-injected-iclock) 7. [07HttpClient instances shared without per-test handlers](#pattern-httpclient-instances-shared-without-per-test-handlers) 8. [08Retry attribute hiding real bugs](#pattern-retry-attribute-hiding-real-bugs) Pattern 1 ### Parallelizable scope confusion across fixtures Symptom. Tests that pass in serial mode start failing intermittently after \`\[assembly: Parallelizable(ParallelScope.Children)\]\` is enabled, with assertions about state another fixture mutated. Root cause. [`[Parallelizable]`](https://docs.nunit.org/articles/nunit/writing-tests/attributes/parallelizable.html) takes a scope (`Self`, `Children`, `Fixtures`, `All`, `None`). The scopes compose by aggregation: assembly-level `Children` + fixture-level `All` means tests in that fixture run alongside tests in every other fixture too. Anything those threads share (a static field, a singleton service, an environment variable) becomes a race. ``` [assembly: Parallelizable(ParallelScope.Fixtures)] public class FeatureFlagTests { public static bool _enabled; [SetUp] public void Setup() { _enabled = true; } [Test] public void NewBillingPathIsActive() { Assert.That(_enabled, Is.True); } } public class LegacyBillingTests { [Test] public void DefaultsToLegacyPath() { // FeatureFlagTests races ahead and sets _enabled = true; assertion fails Assert.That(FeatureFlagTests._enabled, Is.False); } } ``` Fix. Pick parallelism explicitly. Annotate fixtures that share static state with `[NonParallelizable]`, or refactor to remove the static state. For per-thread state, use `AsyncLocal` or `ThreadLocal` rather than plain statics. ``` [NonParallelizable] public class FeatureFlagTests { private bool _enabled; [SetUp] public void Setup() { _enabled = true; } [Test] public void NewBillingPathIsActive() { Assert.That(_enabled, Is.True); } } ``` With Mergify. Test Engine tags failures that only appear under parallel runs as parallelism-sensitive. The dashboard surfaces the parallel-only signature so the shared-static root cause is the obvious lead. Pattern 2 ### Static field state surviving the test run Symptom. A test that mutates a static config dictionary passes alone and breaks an unrelated fixture with a value the second test never set. Root cause. Even without parallelism, static fields live for the AppDomain. NUnit creates a fresh instance of each test fixture per test by default, but the static state survives. A test that sets `FeatureFlags.Enabled["x"] = true` leaves it set for every test that runs after, regardless of fixture or class. ``` public static class FeatureFlags { public static Dictionary Enabled { get; } = new(); } public class FeatureFlagTests { [Test] public void NewBilling() { FeatureFlags.Enabled["NEW_BILLING"] = true; Assert.That(FeatureFlags.Enabled["NEW_BILLING"], Is.True); } } public class PricingTests { [Test] public void DefaultPricing() { // expects Enabled empty; sees ["NEW_BILLING" => true] from sibling Assert.That(Pricing.For("pro"), Is.EqualTo(99)); } } ``` Fix. Reset static state in `[TearDown]` on a base test class so every test starts clean. Better: refactor the static into an injected service so each test owns its own instance. ``` public abstract class TestBase { [TearDown] public void ResetGlobalState() { FeatureFlags.Enabled.Clear(); } } public class FeatureFlagTests : TestBase { [Test] public void NewBilling() { ... } } ``` With Mergify. Test Engine catches the cross-test signature: a test only fails after a specific other test has run, with assertions about a value the failing test never set. The dashboard surfaces the predecessor so the leaking static is the obvious lead. Pattern 3 ### SetUpFixture lifecycle surprises Symptom. A \`\[SetUpFixture\]\` initializes a database container; a test failure leaves the container running because \`\[OneTimeTearDown\]\` did not fire. Root cause. [`[SetUpFixture]`](https://docs.nunit.org/articles/nunit/writing-tests/attributes/setupfixture.html) runs once per namespace before any test in that namespace, with paired `[OneTimeSetUp]` and `[OneTimeTearDown]` hooks. Teardown only fires on graceful shutdown; if the test runner crashes, the host process is killed, or a previous teardown threw, external resources stay open. CI runs that container forever until the next pipeline kills it, and meanwhile the next run hits "port already allocated". ``` [SetUpFixture] public class GlobalFixtures { private static IContainer _postgres; [OneTimeSetUp] public async Task Setup() { _postgres = new PostgreSqlBuilder().Build(); await _postgres.StartAsync(); } [OneTimeTearDown] public async Task TearDown() { // never runs if the host process is killed mid-suite await _postgres.DisposeAsync(); } } ``` Fix. Wrap teardown in a defensive try/catch and register a synchronous `AppDomain.ProcessExit` handler that blocks on async disposal so cleanup is deterministic during shutdown. For containers, use Testcontainers' `.WithCleanUp(true)` so the Docker daemon GCs them even if your handler does not run. ``` [OneTimeSetUp] public async Task Setup() { _postgres = new PostgreSqlBuilder().WithCleanUp(true).Build(); await _postgres.StartAsync(); AppDomain.CurrentDomain.ProcessExit += OnProcessExit; } private static void OnProcessExit(object? sender, EventArgs e) { if (_postgres is null) return; try { _postgres.DisposeAsync().AsTask().GetAwaiter().GetResult(); } catch { // best-effort during process shutdown } } ``` With Mergify. Test Engine notices that the first test of a CI run fails with \`port already allocated\` only when the previous run's \`\[OneTimeTearDown\]\` did not log its completion. The dashboard surfaces the cross-run signature so the missing cleanup is the obvious place to look. Pattern 4 ### TestContext attribute leakage between tests Symptom. A test that reads \`TestContext.CurrentContext.Test.Properties\["sessionId"\]\` passes alone and fails inside the suite, with the wrong session set by a sibling fixture. Root cause. [`TestContext.CurrentContext.Test.Properties`](https://docs.nunit.org/articles/nunit/writing-tests/TestContext.html) stores attributes per test, but `TestContext.Out` and `TestContext.WriteLine` share the test runner's writer. Tests that mutate context-level properties via reflection or rely on inherited `[Property]` attributes from a base class can see one another's values when fixture order changes. ``` [TestFixture] public class CheckoutTests { [Test, Property("sessionId", "sess-1")] public void BuysFromUserOne() { var id = TestContext.CurrentContext.Test.Properties.Get("sessionId"); // Test runner shares property bag during parallel runs; // sibling test's property sometimes wins AssertCheckout(id); } } ``` Fix. Pass session-like values through method arguments (`TestCase` data) or instance fields scoped to the fixture instance. Reserve `TestContext.Properties` for read-only metadata declared statically on the test. ``` public class CheckoutTests { private string _sessionId; [SetUp] public async Task SignIn() { _sessionId = await Auth.LoginAsync("user"); } [Test] public void BuysFromUserOne() { AssertCheckout(_sessionId); } } ``` With Mergify. Test Engine catches the cross-test signature: a test only fails after a specific other test has run, with assertions about a value the failing test never set. The dashboard surfaces the predecessor so the misused TestContext is the obvious lead. Pattern 5 ### Async tests that .Result-deadlock on the sync context Symptom. A test that calls \`someAsyncOperation.Result\` passes locally and hangs in CI until the runner kills it as a timeout. Root cause. Pre-.NET 5, the WPF and ASP.NET Classic synchronization contexts captured the calling thread and resumed continuations on it. [Calling `.Result` on a Task](https://devblogs.microsoft.com/pfxteam/should-i-expose-asynchronous-wrappers-for-synchronous-methods/) blocks the calling thread; if the Task wants to resume on that thread to finish, it deadlocks. NUnit's runner does not always install a sync context, but tests that pull one in (UI tests, custom `SynchronizationContext` in `[SetUp]`) trigger the deadlock under load. ``` [Test] public void GetsUserSynchronously() { SynchronizationContext.SetSynchronizationContext(new TestSyncContext()); var user = userService.GetAsync(1).Result; // deadlocks Assert.That(user.Name, Is.EqualTo("Rémy")); } ``` Fix. Make the test method `async Task` and `await` the call. NUnit awaits async test methods natively. Avoid mixing sync waits with async code in tests. ``` [Test] public async Task GetsUser() { var user = await userService.GetAsync(1); Assert.That(user.Name, Is.EqualTo("Rémy")); } ``` With Mergify. Test Engine groups timeout failures by their cause. When a test consistently times out at exactly the runner's wall-clock limit and never produces output, the dashboard tags it as a deadlock candidate so the .Result call is the obvious place to look. Pattern 6 ### DateTime.UtcNow without an injected IClock Symptom. A test that asserts an event happened "within the last second" passes locally and fails on the slower CI runner with a timestamp 1.2 seconds old. Root cause. Calling `DateTime.UtcNow` directly inside production code reads the system clock. A test that asserts on a freshly created timestamp races against scheduler jitter. Locally the assertion fires within a millisecond; on CI the same code runs after a longer pause and the assertion's tolerance window is too tight. ``` public class AuditLogger { public void Log(string evt) { store[evt] = DateTime.UtcNow; // direct clock read } } [Test] public void AuditTimestampIsRecent() { auditLogger.Log("checkout"); var logged = store["checkout"]; Assert.That(logged, Is.EqualTo(DateTime.UtcNow).Within(TimeSpan.FromSeconds(1))); // racy under CI scheduler jitter } ``` Fix. Inject [`TimeProvider`](https://learn.microsoft.com/dotnet/api/system.timeprovider) (built-in since .NET 8) into the production class so tests can swap in a fixed clock. Production wires `TimeProvider.System`; tests use `FakeTimeProvider` from the [Microsoft.Extensions.TimeProvider.Testing](https://learn.microsoft.com/dotnet/api/microsoft.extensions.time.testing.faketimeprovider) package, or a small custom subclass of `TimeProvider` when adding the dependency is overkill. ``` public class AuditLogger { private readonly TimeProvider _clock; public AuditLogger(TimeProvider clock) { _clock = clock; } public void Log(string evt) { store[evt] = _clock.GetUtcNow(); } } [Test] public void AuditTimestamp() { var fakeTime = new FakeTimeProvider(DateTimeOffset.Parse("2026-01-01Z")); var logger = new AuditLogger(fakeTime); logger.Log("checkout"); Assert.That(store["checkout"], Is.EqualTo(fakeTime.GetUtcNow())); } ``` With Mergify. Test Engine links timing failures to their CI runner type. When a test only fails on the slower runner pool and never on the laptop pool, the dashboard surfaces the resource sensitivity so the real-clock dependency is the obvious place to look. Pattern 7 ### HttpClient instances shared without per-test handlers Symptom. A test that mocks an HTTP response passes alone and fails inside the suite when a sibling test sees the cached mock response from the previous test. Root cause. `HttpClient` is meant to be reused across an application's lifetime: instance-per-test creates socket exhaustion. A common pattern is to register a single `HttpClient` in DI with a custom `DelegatingHandler`. Tests that swap the handler to a mock without resetting after leak the mock into the next test. ``` public class HttpClientTests { private static readonly MockHandler Handler = new(); private static readonly HttpClient Shared = new(Handler); [Test] public async Task UsesMockResponse() { Handler.EnqueueResponse("{}"); var resp = await Shared.GetAsync("https://api.test/me"); Assert.That(await resp.Content.ReadAsStringAsync(), Is.EqualTo("{}")); } [Test] public async Task ReturnsRealValue() { // expects unmocked. actual: mock from previous test still set. var resp = await Shared.GetAsync("https://api.test/me"); Assert.That(await resp.Content.ReadAsStringAsync(), Is.Not.EqualTo("{}")); } } ``` Fix. Build a fresh `HttpClient` with a per-test handler in `[SetUp]`. For shared infrastructure, use `IHttpClientFactory` with named clients so each test asks for its own. ``` public class HttpClientTests { private MockHandler _handler; private HttpClient _client; [SetUp] public void SetUp() { _handler = new MockHandler(); _client = new HttpClient(_handler); } [TearDown] public void TearDown() => _client.Dispose(); [Test] public async Task UsesMockResponse() { _handler.EnqueueResponse("{}"); var resp = await _client.GetAsync("https://api.test/me"); Assert.That(await resp.Content.ReadAsStringAsync(), Is.EqualTo("{}")); } } ``` With Mergify. Test Engine groups failures whose only signature is unexpected HTTP responses or mismatched assertion strings into a per-suite bucket. The dashboard surfaces the test that first poisoned the shared client so the fix lands at the source. Pattern 8 ### Retry attribute hiding real bugs Symptom. Your suite is green. A user reports a bug that should have been caught by the test that ran three times yesterday before passing on attempt 3. Root cause. [`[Retry(3)]`](https://docs.nunit.org/articles/nunit/writing-tests/attributes/retry.html) reruns failing tests up to N times and reports the last result. A real race that loses on attempt 1 and wins on attempt 2 gets reported as PASSED. The bug is still there. The pipeline has decided not to look at it. ``` [Test, Retry(3)] public async Task ChargesCard() { // intermittent timing bug; passes 2 of 3 times var result = await billingService.ChargeAsync(42); Assert.That(result.Success, Is.True); } ``` Fix. Do not retry at the framework level. When a test is genuinely flaky, fix it. When the fix takes longer than a session, quarantine it instead. That keeps the signal visible without blocking the merge queue. With Mergify. Test Engine reruns at the CI level with attempt-level result tracking. You see that a test passed on attempt 2 of 3, which is exactly the information \`\[Retry\]\` throws away. Quarantine kicks in once the pattern is clear. Detection ## How to detect flaky NUnit tests in CI NUnit's \`dotnet test\` runner emits TRX or JUnit XML reports depending on the logger. Configure the JUnit logger and upload the resulting XML to Mergify with a one-line CLI call. Test Engine builds a confidence score for every test on your default branch. PR runs are compared against that baseline. Anything inconsistent gets flagged in a PR comment before the author merges. mergify ci ``` # 1. Add the JUnit logger dotnet add package JUnitTestLogger # 2. Emit JUnit XML on every CI run dotnet test --logger "junit;LogFilePath=junit.xml" # 3. Upload the result (once, in CI) curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh mergify ci junit-process junit.xml ``` Prevention ## How to prevent flaky NUnit tests from merging On every PR, Mergify reruns the tests whose confidence is below threshold, without \[Retry\] touching your config. The PR gets a comment naming the unreliable tests, their confidence history, and whether the failure on this PR is new or historical noise. Authors fix the real bugs before merge instead of re-running CI until it passes. Quarantine ## How to quarantine flaky NUnit tests without skipping them Once a NUnit test is confirmed flaky, Test Engine quarantines it. The test still runs in the suite, no \`\[Ignore\]\` rewrite required, but its result no longer blocks merges or marks the pipeline red. When the pass rate on main recovers, quarantine lifts automatically and the test goes back to being load-bearing. Your CI ## Run it on your CI The NUnit setup for Test Engine on each CI, how its retries compare with the CI's own reruns, and what makes NUnit flake on that CI's runners. - [NUnit on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions/nunit) ## Want to see which NUnit tests in your repo are already flaky? Works with the JUnit logger or any JUnit-compatible NUnit reporter. Setup takes under five minutes. [Book a discovery call](https://mergify.com/discovery) ## Frequently asked questions Why are my NUnit tests flaky in CI but pass locally? Your laptop and your CI runner differ in CPU count, parallel scope settings, and whether a sibling test mutated a static field before yours ran. NUnit tests that share static state across fixtures, deadlock on .Result inside a sync context, or read DateTime.UtcNow surface those issues only under CI's tighter resource budget. Reproduce locally with the same NUnit parallelism settings your CI uses (\`LevelOfParallelism\` or \`NumberOfTestWorkers\`, configured via assembly attributes or \`.runsettings\`) to expose the failure, then fix the underlying coupling before pushing. How do I detect flaky NUnit tests? NUnit alone cannot tell flaky from broken since each run gives one data point per test. You need to run the same commit multiple times and compare results. Mergify Test Engine does that on every PR and on the default branch, scores each test, and surfaces the tests whose pass rate drops below a confidence threshold. Does \[Retry\] fix flaky tests? No, it hides them. A test that fails on attempt 1 and passes on attempt 2 is still broken; you have only decided not to look at the failure. Use \[Retry\] as a temporary bandage for a test you are actively fixing, never as a permanent policy. For visibility without blocking the merge queue, quarantine instead of retry. What causes flaky tests in NUnit? Eight patterns cover most of what we see: Parallelizable scope confusion across fixtures, static field state surviving the test run, SetUpFixture lifecycle surprises, TestContext attribute leakage between tests, async tests that .Result-deadlock on the sync context, DateTime.UtcNow without an injected IClock, HttpClient instances shared without per-test handlers, and Retry attribute hiding real bugs. Each is covered above with a minimal reproducer. How do I quarantine a flaky NUnit test without deleting it? Mergify Test Engine quarantines the test automatically once its confidence score drops. The test still runs in the suite, but a failing result no longer blocks merges and its noise no longer drowns out real signal. When the test stabilizes on main, quarantine lifts automatically. No \`\[Ignore\]\`, no commented-out tests, no orphaned files. What's the difference between NUnit ParallelScope.Self and Children? \`Self\` allows the annotated test or fixture to run in parallel with siblings at the same level. \`Children\` allows the annotated container's children to run in parallel with each other. They compose: a fixture marked \`Self\` whose tests are marked \`Self\` runs the fixture in parallel with siblings AND runs the tests in parallel within the fixture. For a fixture that holds shared state, mark it \`\[NonParallelizable\]\` to opt out entirely. ## Ship your NUnit suite green. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/pest [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/Pest # Flaky tests in Pest. Named, fixed, and quarantined. Flaky Pest suites are not random. They follow patterns: higher-order chains sharing state, beforeEach scope confusion, parallel paratest workers fighting over the database, snapshot races, Carbon clocks left frozen. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Published April 2026 Example PR comment from the Mergify bot detecting a flaky Pest test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky Pest suite: higher-order chains that share state across iterations, beforeEach mutating global state without afterEach cleanup, parallel paratest workers fighting over the Laravel database, dataset() closures capturing mutable outer state, snapshot-driven tests racing in parallel, Carbon test-time leakage between specs, browser tests via Pest Plugin Browser racing the page, and Pest's retry expression hiding real bugs. Each one has a minimal reproducer, a fix in stock Pest, and a Mergify Test Engine detection strategy below. ## Why are Pest tests flaky? [Pest](https://pestphp.com/) is the modern PHP testing framework built on top of PHPUnit. It keeps PHPUnit's runner and trades the class-based syntax for an expressive function-based DSL: `test('does the thing', fn () => ...)`, higher-order chains, datasets, and architectural assertions. The DSL makes specs read like the behaviour they test. The catch is that everything still runs on PHPUnit underneath, so every PHPUnit flake category applies, plus a few that come from Pest's syntax sugar. Two facets are Pest-specific. [Higher-order tests](https://pestphp.com/docs/higher-order-testing) chain expectations on the test instance, which makes it easy to accidentally share state across the chain. [Datasets](https://pestphp.com/docs/datasets) can be defined as closures or arrays, and a closure that captures mutable state outside its scope leaks across every test that uses it. Pest's parallel runner inherits paratest's database-collision problem when used with Laravel's `RefreshDatabase`. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of Pest suites: higher-order chains that share state across iterations, beforeEach mutating global state without afterEach cleanup, parallel paratest workers fighting over the Laravel database, dataset() closures capturing mutable outer state, snapshot-driven tests racing in parallel, Carbon test-time leakage between specs, browser tests via Pest Plugin Browser racing the page, and Pest's retry expression hiding real bugs. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01Higher-order chains that share state across iterations](#pattern-higher-order-chains-that-share-state-across-iterations) 2. [02beforeEach mutating global state without afterEach cleanup](#pattern-beforeeach-mutating-global-state-without-aftereach-cleanup) 3. [03Parallel paratest workers fighting over the Laravel database](#pattern-parallel-paratest-workers-fighting-over-the-laravel-database) 4. [04dataset() closures capturing mutable outer state](#pattern-dataset-closures-capturing-mutable-outer-state) 5. [05Snapshot-driven tests racing in parallel](#pattern-snapshot-driven-tests-racing-in-parallel) 6. [06Carbon test-time leakage between specs](#pattern-carbon-test-time-leakage-between-specs) 7. [07Browser tests via Pest Plugin Browser racing the page](#pattern-browser-tests-via-pest-plugin-browser-racing-the-page) 8. [08Pest's retry expression hiding real bugs](#pattern-pest-s-retry-expression-hiding-real-bugs) Pattern 1 ### Higher-order chains that share state across iterations Symptom. A higher-order Pest test that expects multiple values on the same subject passes locally and fails in CI when the subject's state changed between expectations. Root cause. [Higher-order tests](https://pestphp.com/docs/higher-order-testing) chain method calls on the test's expected subject. Each link in the chain operates on the same value, so a chain that includes a method call with side effects (a fluent builder, a state machine transition) carries those side effects into the next assertion. Locally the chain runs fast enough that intermediate state matches; on CI a slower scheduler exposes the order dependence. ``` it('renames and saves the user') ->expect(fn () => User::create(['name' => 'Rémy'])) ->name->toBe('Rémy') ->setName('Rémy Updated') // mutates the expected value ->name->toBe('Rémy Updated') ->save() // calls a slow DB write ->name->toBe('Rémy Updated'); // sometimes sees a stale cached value ``` Fix. Use higher-order chains for read-only assertions on a single value. When the test mutates the subject, switch to an explicit closure body so each step's side effects are visible and the order is unambiguous. ``` it('renames and saves the user', function () { $user = User::create(['name' => 'Rémy']); expect($user->name)->toBe('Rémy'); $user->setName('Rémy Updated'); expect($user->name)->toBe('Rémy Updated'); $user->save(); expect($user->fresh()->name)->toBe('Rémy Updated'); }); ``` With Mergify. Test Engine groups higher-order test failures by the assertion position in the chain. When the same chain consistently fails at step 3 of 5, the dashboard surfaces the cross-step state dependency so the side-effect-free rewrite is the obvious fix. Pattern 2 ### beforeEach mutating global state without afterEach cleanup Symptom. A test inside a \`describe\` block changes a Laravel facade fake or a static class property, the next test in a sibling block fails on an assertion about a value the failing test never set. Root cause. [Pest's `beforeEach`](https://pestphp.com/docs/hooks) always fires before every test in scope, but it does not auto-reset what it changed. A hook inside a `describe` that mutates a static property, registers a permanent event listener, or fakes a facade without a paired `afterEach` leaks that change into every later test in the file. The file-level `beforeEach` does not undo it. ``` // tests/Feature/InvoiceTest.php describe('with frozen audit log', function () { beforeEach(function () { AuditLog::$disabled = true; // static property mutation }); test('processes order without audit', function () { expect(processOrder())->toBeTrue(); }); }); // later in the same file, no longer inside the describe test('writes an audit entry for refunds', function () { refund(); // expects AuditLog::$disabled === false (default) // actual: the previous describe set it true and never reset it expect(AuditLog::entries())->toHaveCount(1); }); ``` Fix. Pair every `beforeEach` that mutates global state with an `afterEach` in the same scope so the change is undone before the next test. For Laravel facades, prefer the test traits Laravel ships (RefreshDatabase, WithFaker) that handle teardown for you. ``` describe('with frozen audit log', function () { beforeEach(fn () => AuditLog::$disabled = true); afterEach(fn () => AuditLog::$disabled = false); test('processes order without audit', function () { expect(processOrder())->toBeTrue(); }); }); ``` With Mergify. Test Engine groups failures by the describe block that ran before them. When several seemingly unrelated tests start failing only after a specific describe runs, the dashboard surfaces the upstream culprit so the missing afterEach is the obvious lead. Pattern 3 ### Parallel paratest workers fighting over the Laravel database Symptom. A Pest suite that runs green sequentially fails under \`pest --parallel\` with assertions about rows another worker created or \`SQLSTATE: database is locked\`. Root cause. Pest's parallel runner is paratest under the hood. [RefreshDatabase](https://laravel.com/docs/testing#refreshdatabase) wraps each test in a transaction, but every parallel worker hits the same database file or schema. SQLite locks immediately; Postgres/MySQL race on autoincrement IDs. Pest does not auto-recreate per-worker databases the way `php artisan test --parallel` does unless the runner is configured to. ``` # CI script vendor/bin/pest --parallel --processes=4 # Worker 1 starts a transaction, Worker 2 hits SQLITE_BUSY, suite fails. # .env.testing DB_CONNECTION=sqlite DB_DATABASE=database/testing.sqlite ``` Fix. Use Laravel's `php artisan test --parallel` wrapper around Pest, which handles per-worker databases automatically. For raw paratest invocations, set the database name to a per-token placeholder. ``` # CI script php artisan test --parallel --processes=4 # Or with raw paratest: vendor/bin/pest --parallel --recreate-databases # .env.testing for Postgres DB_CONNECTION=pgsql DB_DATABASE=test_${TEST_TOKEN} ``` With Mergify. Test Engine tags failures that only appear under parallel runs as parallelism-sensitive. The dashboard surfaces the parallel-only signature so the worker-database collision is the obvious root cause. Pattern 4 ### dataset() closures capturing mutable outer state Symptom. A dataset-driven test passes for some inputs and fails for others, and re-running with the failing input alone passes. Root cause. [Pest datasets](https://pestphp.com/docs/datasets) can be defined as closures that build values per iteration. A closure that captures a mutable object from the surrounding scope returns the same instance on every call, so each iteration sees the previous one's mutations. `use ($shared)` in a dataset closure does not give the iteration its own copy. ``` $shared = User::factory()->make(['name' => 'Rémy']); dataset('plans', function () use ($shared) { // closure captures $shared by reference; every yield returns the SAME instance yield 'free' => ['free', $shared]; yield 'pro' => ['pro', $shared]; }); it('renames the user', function (string $plan, User $u) { $u->setName($u->getName() . '-' . $plan); expect($u->name)->toEndWith($plan); // row 1 mutates $shared; row 2 sees 'Rémy-free' and renames to 'Rémy-free-pro' })->with('plans'); ``` Fix. Build fresh values inside the dataset closure on every yield so each iteration gets its own. For expensive setup, factor it into a factory function the dataset calls per row. ``` dataset('plans', function () { yield 'free' => ['free', User::factory()->make(['name' => 'Rémy'])]; yield 'pro' => ['pro', User::factory()->make(['name' => 'Rémy'])]; }); ``` With Mergify. Test Engine groups failures by test name and dataset row. When iteration N of a dataset-driven test fails consistently and N-1 passed, the dashboard surfaces the iteration-order signature so the shared-closure mistake is easy to find. Pattern 5 ### Snapshot-driven tests racing in parallel Symptom. A test using \`toMatchSnapshot()\` writes a snapshot file in CI and the next run fails with a diff that did not exist between commits. Root cause. Snapshot files in [spatie/pest-plugin-snapshots](https://github.com/spatie/pest-plugin-snapshots) live on disk under `tests/__snapshots__`. Under `--parallel`, two workers can write to the same snapshot file simultaneously when two tests share the same descriptor (a parameterised test, two tests with the same name in different describes). The result is one snapshot file with content from whichever write landed last. ``` it('renders invoice in USD', function () { expect(format(1099, 'USD'))->toMatchSnapshot(); }); it('renders invoice in EUR', function () { expect(format(1099, 'EUR'))->toMatchSnapshot(); }); # Both tests can run on different workers and write to: # tests/__snapshots__/InvoiceTest__renders_invoice__1.snap # (the snapshot key collides when the test names share a prefix) ``` Fix. Keep snapshot tests in the same file and either run them serially (group them under a single `describe` with explicit naming) or split snapshot tests into a separate suite that does not run with `--parallel`. ``` // pest.php uses(MatchesSnapshots::class)->in('Snapshots'); // CI: vendor/bin/pest --testsuite=Unit --parallel vendor/bin/pest --testsuite=Snapshots // sequential ``` With Mergify. Test Engine detects the snapshot-churn pattern: the same test file produces a different .snap diff on consecutive runs of the same SHA. The dashboard surfaces the file as snapshot-unstable so the parallel-write race is the obvious lead. Pattern 6 ### Carbon test-time leakage between specs Symptom. A test that calls \`Carbon::setTestNow('2026-01-01')\` passes, and a later spec that touches \`Carbon::now()\` asserts on a date months in the past. Root cause. [Carbon's `setTestNow`](https://github.com/briannesbitt/Carbon) mutates a static. Pest does not auto-reset it between tests. A spec that froze the clock without resetting in `afterEach` leaves the clock frozen for every test that runs after on the same worker. The leak is invisible until a later test reads `Carbon::now()` and fails. ``` it('expires the invitation', function () { Carbon::setTestNow('2026-01-01'); $invite = Invitation::create(); Carbon::setTestNow('2026-01-09'); expect($invite->isExpired())->toBeTrue(); // missing Carbon::setTestNow(); }); it('mints a session token valid for an hour', function () { $token = SessionToken::for($this->user); // Carbon::now() is still 2026-01-09; assertion fails expect($token->expiresAt->timestamp) ->toBeWithin(60, Carbon::now()->addHour()->timestamp); }); ``` Fix. Reset Carbon's test clock in a global `afterEach` in `tests/Pest.php`. Or use the closure form of `Carbon::withTestNow` so the clock auto-restores at the end of the block. ``` // tests/Pest.php afterEach(function () { Carbon::setTestNow(); // reset to real clock }); // in a test it('expires the invitation', function () { Carbon::withTestNow('2026-01-01', function () { $invite = Invitation::create(); Carbon::setTestNow('2026-01-09'); expect($invite->isExpired())->toBeTrue(); }); }); ``` With Mergify. Test Engine catches the cross-test signature: a test only fails after a known time-mutating test, and only when its assertions touch the clock. The dashboard surfaces the ordering so the missed Carbon::setTestNow() is easy to locate. Pattern 7 ### Browser tests via Pest Plugin Browser racing the page Symptom. A \`pestphp/pest-plugin-browser\` (Playwright-driven) test passes locally and fails in CI with \`element not found\` on a button the screenshot clearly shows. Root cause. [pest-plugin-browser](https://github.com/pestphp/pest-plugin-browser) wraps Playwright and inherits its auto-wait semantics. A test that sleeps for a hardcoded duration before clicking races the page exactly the way bare Playwright does. Locally the wait covers the render; on the slower CI runner the click fires before the element is interactable. ``` it('opens the modal', function () { visit('/dashboard'); click('button.open-modal'); sleep(1); // hope the modal is open type('input[name=email]', 'user@example.com'); // CI: input not found, the modal animation is still running }); ``` Fix. Wait for the actual signal: an assertion that the modal is visible. The Pest browser plugin's `assertVisible` retries with the configured timeout, so the test is fast when fast and patient when slow. ``` it('opens the modal', function () { visit('/dashboard'); click('button.open-modal'); assertVisible('[role=dialog]'); type('[role=dialog] input[name=email]', 'user@example.com'); }); ``` With Mergify. Test Engine links browser-test failures to their CI runner type. When a Pest browser test only fails on the slower runner pool and never on the laptop pool, the dashboard surfaces the resource sensitivity so the hardcoded sleep is the obvious place to look. Pattern 8 ### Pest's retry expression hiding real bugs Symptom. Your Pest suite is green. A user reports a bug that should have been caught by the test that ran three times yesterday before passing. Root cause. [Pest supports retries](https://pestphp.com/docs/cli-api-reference) via `--retry` on the CLI or per-test `->retry()`. A real race that loses on attempt 1 and wins on attempt 2 gets reported as PASSED. The bug is still there. The pipeline has decided not to look at it. ``` // CI script (please don't) vendor/bin/pest --retry // or per test (please don't) it('charges the card', function () { // intermittent timing bug })->retry(3); ``` Fix. Do not retry at the framework level. When a test is genuinely flaky, fix it. When the fix takes longer than a session, quarantine it instead. That keeps the signal visible without blocking the merge queue. With Mergify. Test Engine reruns at the CI level with attempt-level result tracking. You see that a test passed on attempt 2 of 3, which is exactly the information \`--retry\` and \`->retry()\` throw away. Quarantine kicks in once the pattern is clear. Detection ## How to detect flaky Pest tests in CI Pest emits JUnit-compatible XML reports with the \`--log-junit\` flag (PHPUnit's option, inherited). Point Mergify at the XML output of every CI run and Test Engine builds a confidence score for every test on your default branch. PR runs are compared against that baseline. Anything inconsistent gets flagged in a PR comment before the author merges. mergify ci ``` # 1. Emit JUnit XML on every CI run vendor/bin/pest --log-junit junit.xml # Or via Laravel's wrapper: php artisan test --log-junit=junit.xml # 2. Upload the result (once, in CI) curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh mergify ci junit-process junit.xml ``` Prevention ## How to prevent flaky Pest tests from merging On every PR, Mergify reruns the tests whose confidence is below threshold, without Pest's \`--retry\` flag or \`->retry()\` touching your config. The PR gets a comment naming the unreliable tests, their confidence history, and whether the failure on this PR is new or historical noise. Authors fix the real bugs before merge instead of re-running CI until it passes. Quarantine ## How to quarantine flaky Pest tests without skipping them Once a Pest test is confirmed flaky, Test Engine quarantines it. The test still runs in the suite, no \`$this->markTestSkipped()\` rewrite required, but its result no longer blocks merges or marks the pipeline red. When the pass rate on main recovers, quarantine lifts automatically and the test goes back to being load-bearing. Your CI ## Run it on your CI The Pest setup for Test Engine on each CI, how its retries compare with the CI's own reruns, and what makes Pest flake on that CI's runners. - [Pest on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions/pest) ## Want to see which Pest tests in your repo are already flaky? Works with Pest's built-in \`--log-junit\` output. No extra plugins required. Setup takes under five minutes. [Book a discovery call](https://mergify.com/discovery) ## Frequently asked questions Why are my Pest tests flaky in CI but pass locally? Your laptop and your CI runner differ in CPU count, parallel paratest worker count, and whether the worker shares a database with siblings. Pest tests that share state across higher-order chains, mutate Carbon's clock without resetting, or depend on hardcoded sleeps in browser specs surface those issues only under CI's tighter resource budget. Reproduce locally with \`php artisan test --parallel --processes=4\` to push the failure into the open before pushing. How do I detect flaky Pest tests? Pest alone cannot tell flaky from broken since each run gives one data point per test. You need to run the same commit multiple times and compare results. Mergify Test Engine does that on every PR and on the default branch, scores each test, and surfaces the tests whose pass rate drops below a confidence threshold. Does Pest's \`--retry\` flag or \`->retry()\` fix flaky tests? No, it hides them. A test that fails on attempt 1 and passes on attempt 2 is still broken; you have only decided not to look at the failure. Use Pest's \`--retry\` flag or \`->retry()\` as a temporary bandage for a test you are actively fixing, never as a permanent policy. For visibility without blocking the merge queue, quarantine instead of retry. What causes flaky tests in Pest? Eight patterns cover most of what we see: higher-order chains that share state across iterations, beforeEach mutating global state without afterEach cleanup, parallel paratest workers fighting over the Laravel database, dataset() closures capturing mutable outer state, snapshot-driven tests racing in parallel, Carbon test-time leakage between specs, browser tests via Pest Plugin Browser racing the page, and Pest's retry expression hiding real bugs. Each is covered above with a minimal reproducer. How do I quarantine a flaky Pest test without deleting it? Mergify Test Engine quarantines the test automatically once its confidence score drops. The test still runs in the suite, but a failing result no longer blocks merges and its noise no longer drowns out real signal. When the test stabilizes on main, quarantine lifts automatically. No \`$this->markTestSkipped()\`, no commented-out tests, no orphaned files. What's the difference between Pest higher-order tests and explicit closures? Higher-order tests chain expectations on the test's expected subject in a fluent style: \`expect(fn () => User::find(1))->name->toBe('Rémy')\`. Explicit closures pass an anonymous function as the test body and set up state inline. Higher-order is concise for read-only assertions on a single value; explicit closures are unambiguous when the test mutates state across multiple steps. Mixing the two in one test is the trap, because a fluent chain with side effects is hard to reason about under parallelism. ## Ship your Pest suite green. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/phpunit [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/PHPUnit # Flaky tests in PHPUnit. Named, fixed, and quarantined. Flaky PHPUnit suites are not random. They follow patterns: static property leakage, paratest workers fighting over the database, dataProvider rows mutating shared state, Mockery expectations unclosed, Carbon clocks left frozen. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Published April 2026 Example PR comment from the Mergify bot detecting a flaky PHPUnit test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky PHPUnit suite: static property leakage between tests, RefreshDatabase against parallel paratest workers, dataProvider returning shared mutable state, Mockery expectations leaking across tests, Carbon::now() without setTestNow, Guzzle MockHandler queues that outlive their test, processIsolation traps for global state, and PHPUnit retry attribute hiding real bugs. Each one has a minimal reproducer, a fix in stock PHPUnit, and a Mergify Test Engine detection strategy below. ## Why are PHPUnit tests flaky? [PHPUnit](https://phpunit.de/) runs every test method on a fresh instance of the test class by default, which gives you per-method isolation for free. The catch is that PHP itself does not isolate between tests in the same process: globals, static properties, the autoloader's cache, and any singleton stay alive for the life of the worker. PHPUnit gives you escape hatches (`processIsolation`, `@runInSeparateProcess`) but they are off by default because they are slow. Layer on the Laravel ecosystem, where most PHPUnit suites live in 2026. [RefreshDatabase](https://laravel.com/docs/testing#refreshdatabase) wraps each test in a transaction that rolls back, but parallel [paratest](https://github.com/paratestphp/paratest) workers fight over the same database unless you give each worker its own. [Mockery](https://github.com/Mockery/mockery) expectations need explicit closing. [Carbon](https://github.com/briannesbitt/Carbon)'s `setTestNow` mutates a static and stays set unless you reset it. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of PHPUnit suites: static property leakage between tests, RefreshDatabase against parallel paratest workers, dataProvider returning shared mutable state, Mockery expectations leaking across tests, Carbon::now() without setTestNow, Guzzle MockHandler queues that outlive their test, processIsolation traps for global state, and PHPUnit retry attribute hiding real bugs. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01Static property leakage between tests](#pattern-static-property-leakage-between-tests) 2. [02RefreshDatabase against parallel paratest workers](#pattern-refreshdatabase-against-parallel-paratest-workers) 3. [03dataProvider returning shared mutable state](#pattern-dataprovider-returning-shared-mutable-state) 4. [04Mockery expectations leaking across tests](#pattern-mockery-expectations-leaking-across-tests) 5. [05Carbon::now() without setTestNow](#pattern-carbon-now-without-settestnow) 6. [06Guzzle MockHandler queues that outlive their test](#pattern-guzzle-mockhandler-queues-that-outlive-their-test) 7. [07processIsolation traps for global state](#pattern-processisolation-traps-for-global-state) 8. [08PHPUnit retry attribute hiding real bugs](#pattern-phpunit-retry-attribute-hiding-real-bugs) Pattern 1 ### Static property leakage between tests Symptom. A test that mutates a \`static\` property on a service class passes alone and breaks an unrelated test class with a value the second test never set. Root cause. PHPUnit creates a fresh instance of the test class per method, but PHP `static` properties live for the life of the worker. A test that calls `FeatureFlag::set('NEW_BILLING', true)` leaves the flag set for every later test in the run. The autoloader's cache, registry singletons, and any class with a static collection are vulnerable to the same leak. ``` class FeatureFlag { public static array $flags = []; public static function set(string $k, bool $v): void { self::$flags[$k] = $v; } } class FeatureFlagTest extends TestCase { public function testNewBillingPath(): void { FeatureFlag::set('NEW_BILLING', true); $this->assertTrue(FeatureFlag::$flags['NEW_BILLING']); } } class LegacyBillingTest extends TestCase { public function testCharge(): void { // expects FeatureFlag::$flags empty; actual: ['NEW_BILLING' => true] leaked $this->assertSame(99, Pricing::for('pro')); } } ``` Fix. Reset static state in `tearDown()`, or use the `@backupStaticAttributes` annotation (slow). The cleanest fix is to avoid static mutable state in tested classes; pass configuration through a service container instead. ``` class FeatureFlagTest extends TestCase { protected function tearDown(): void { FeatureFlag::$flags = []; parent::tearDown(); } public function testNewBillingPath(): void { FeatureFlag::set('NEW_BILLING', true); $this->assertTrue(FeatureFlag::$flags['NEW_BILLING']); } } ``` With Mergify. Test Engine catches the cross-test signature: a test only fails after a specific other test has run, with assertions about a value the failing test never set. The dashboard surfaces the predecessor so the leaking static is the obvious lead. Pattern 2 ### RefreshDatabase against parallel paratest workers Symptom. A Laravel test suite that runs green sequentially fails under \`paratest\` with \`SQLSTATE: database is locked\` or assertions about rows that another worker created. Root cause. [RefreshDatabase](https://laravel.com/docs/testing#refreshdatabase) wraps each test in a transaction. With one worker that works fine. [paratest](https://github.com/paratestphp/paratest) spawns N workers, all hitting the same database file or schema. SQLite locks immediately; Postgres/MySQL race on autoincrement IDs. Laravel ships `--recreate-databases` to fix this but you have to enable it. ``` // phpunit.xml # CI script vendor/bin/paratest -p 4 # Worker 1 transaction holds, Worker 2 hits SQLITE_BUSY, suite fails. ``` Fix. Run `paratest --recreate-databases` so each worker gets a per-process database (or schema, depending on the driver). For Postgres/MySQL, set `DB_DATABASE_TEST` with a placeholder Laravel substitutes per worker. ``` # CI script vendor/bin/paratest -p 4 --recreate-databases --runner=WrapperRunner # .env.testing DB_CONNECTION=pgsql DB_DATABASE=test_${TEST_TOKEN} ``` With Mergify. Test Engine tags failures that only appear under parallel runs as parallelism-sensitive. The dashboard surfaces the parallel-only signature so the worker-database collision is the obvious root cause. Pattern 3 ### dataProvider returning shared mutable state Symptom. A data-driven test passes for some inputs and fails for others, and re-running with the failing input alone passes. Root cause. [`@dataProvider`](https://docs.phpunit.de/en/10.5/writing-tests-for-phpunit.html#data-providers) returns an iterable of arguments that PHPUnit calls the test with once per row. If the rows hold mutable objects, every iteration mutates the same instance. Row 0 changes a shared `User`, row 1 sees the post-row-0 state. ``` class PricingTest extends TestCase { private static User $sharedUser; public static function setUpBeforeClass(): void { self::$sharedUser = new User(name: 'Rémy'); } public static function plans(): array { return [['free', self::$sharedUser], ['pro', self::$sharedUser]]; } /** @dataProvider plans */ public function testRenamesUser(string $plan, User $u): void { $u->setName($u->getName() . '-' . $plan); $this->assertStringEndsWith($plan, $u->getName()); // row 1: 'Rémy-free' renamed to 'Rémy-free-pro'; assertion still passes // row 2 might run first under parallel: name is wrong by the time row 1 reads } } ``` Fix. Build fresh instances inside the provider. PHPUnit calls the provider once and iterates the result, so building inside the iteration body keeps each row independent. ``` public static function plans(): array { return [ ['free', new User(name: 'Rémy')], ['pro', new User(name: 'Rémy')], ]; } ``` With Mergify. Test Engine groups failures by test method and parameter index. When iteration N of a data-provider test fails consistently and N-1 passed, the dashboard surfaces the iteration-order signature so the shared-state mistake is easy to find. Pattern 4 ### Mockery expectations leaking across tests Symptom. A test sets a Mockery expectation, the assertion passes, and a sibling test fails on a method that was never declared in its own scope. Root cause. [Mockery](https://github.com/Mockery/mockery) stores expectations on a global container. `Mockery::close()` in `tearDown` verifies and clears them, but tests that skip `tearDown` (an exception in setUp, an early return) leave expectations behind. The next test sees a mock with stubbed behavior it never declared. ``` class BillingTest extends TestCase { public function testChargesUser(): void { $stripe = Mockery::mock(StripeClient::class); $stripe->shouldReceive('charge')->andReturn(true); // exits without Mockery::close(); expectation persists $this->assertTrue(Billing::for($stripe)->chargeUser(42)); } public function testFallback(): void { $client = new StripeClient(); // sees the global Mockery container with the leftover expectation $this->assertTrue(Billing::for($client)->fallbackFor(42)); } } ``` Fix. Use `Mockery\Adapter\Phpunit\MockeryPHPUnitIntegration` (a trait) so `Mockery::close()` runs automatically after every test, even on failure. For Laravel, the trait is included in the base `TestCase`. ``` use Mockery\Adapter\Phpunit\MockeryPHPUnitIntegration; class BillingTest extends TestCase { use MockeryPHPUnitIntegration; // close() fires in tearDown automatically public function testChargesUser(): void { $stripe = Mockery::mock(StripeClient::class); $stripe->shouldReceive('charge')->andReturn(true); $this->assertTrue(Billing::for($stripe)->chargeUser(42)); } } ``` With Mergify. Test Engine catches the cross-test signature: a test only fails when run after a specific other test, and only when the assertion involves a Mockery-managed dependency. The dashboard tags the dependency so you know it's mock leakage, not a real regression. Pattern 5 ### Carbon::now() without setTestNow Symptom. A test that calls \`Carbon::setTestNow('2026-01-01')\` passes, and the next test that touches \`Carbon::now()\` fails with a date months in the past or future. Root cause. [`Carbon::setTestNow`](https://github.com/briannesbitt/Carbon) mutates a static on the Carbon class. Without a paired `Carbon::setTestNow(null)` in `tearDown`, the frozen time persists for every test that touches `Carbon::now()` on the same worker. Tests that were green for months go red after the first test that froze time without restoring it. ``` class InvitationTest extends TestCase { public function testExpiresAfterSevenDays(): void { Carbon::setTestNow('2026-01-01'); $invite = Invitation::create(); Carbon::setTestNow(Carbon::now()->addDays(8)); $this->assertTrue($invite->isExpired()); // missing Carbon::setTestNow(null); } } class SessionTest extends TestCase { public function testTokenExpiresInOneHour(): void { $token = SessionToken::for($this->user); // Carbon::now() is still January 9 2026; assertion fails $this->assertEqualsWithDelta( Carbon::now()->addHour()->timestamp, $token->expiresAt->timestamp, 60 ); } } ``` Fix. Reset Carbon's test clock in `tearDown()` on a base test case so every test starts with the real clock. For one-off freezes, prefer the closure form that auto-restores. ``` abstract class TestCase extends BaseTestCase { protected function tearDown(): void { Carbon::setTestNow(); // reset to real clock parent::tearDown(); } } // or use the closure form: Carbon::setTestNow('2026-01-01', function () { $invite = Invitation::create(); Carbon::setTestNow(Carbon::now()->addDays(8)); $this->assertTrue($invite->isExpired()); }); ``` With Mergify. Test Engine shows the cross-test time signature: a test only fails after a known time-mutating test, and only when its assertions touch the clock. The dashboard surfaces the ordering so the missed setTestNow(null) is easy to locate. Pattern 6 ### Guzzle MockHandler queues that outlive their test Symptom. A test that pushes responses onto a Guzzle MockHandler passes, and a later test fails with a response from the previous test's queue. Root cause. [Guzzle's MockHandler](https://docs.guzzlephp.org/en/stable/testing.html) is a queue. A test that calls `$handler->append(new Response(200, [], '{}'))` three times and only consumes two leaves one queued. If the handler is shared via a service container, the next test that resolves the same client pulls the leftover response instead of making a real call. ``` class HttpClientTest extends TestCase { public function testFetchesUser(): void { $handler = new MockHandler([new Response(200, [], json_encode(['name' => 'Rémy']))]); $client = new Client(['handler' => HandlerStack::create($handler)]); // app()->instance(Client::class, $client); // bound for the whole test run $this->assertSame('Rémy', User::fetch()->name); } public function testHandles404(): void { // expected: a fresh client; actual: same MockHandler with stale queue $this->expectException(NotFoundHttpException::class); User::fetch(); } } ``` Fix. Build a fresh handler per test inside `setUp` and bind it explicitly. Avoid sharing the handler instance across tests through the service container. ``` class HttpClientTest extends TestCase { private MockHandler $handler; protected function setUp(): void { parent::setUp(); $this->handler = new MockHandler(); app()->instance(Client::class, new Client([ 'handler' => HandlerStack::create($this->handler), ])); } public function testFetchesUser(): void { $this->handler->append(new Response(200, [], json_encode(['name' => 'Rémy']))); $this->assertSame('Rémy', User::fetch()->name); } } ``` With Mergify. Test Engine groups failures whose only signature is unexpected HTTP responses or network exceptions into a per-suite bucket. The dashboard surfaces the test that first poisoned the shared handler so the fix lands at the source. Pattern 7 ### processIsolation traps for global state Symptom. A test class with \`@runInSeparateProcess\` passes locally and fails in CI with a \`serialize()\` error or a missing service-container binding. Root cause. [`@runInSeparateProcess`](https://docs.phpunit.de/en/10.5/annotations.html#runinseparateprocess) forks a fresh PHP process per test for full isolation. PHPUnit serializes test state across the process boundary, but anonymous classes, closures with non-serializable bindings, and database connections cannot cross. A test that captures any of those in `setUp` fails on the serialize step in subtle ways. ``` /** @runInSeparateProcess */ class ConfigTest extends TestCase { private \Closure $factory; protected function setUp(): void { // captures the test instance (\$this) which holds a PDO connection $this->factory = fn (string $k) => Config::get($k); } public function testReadsConfig(): void { $this->assertSame('value', ($this->factory)('key')); // SerializationException: closure cannot be serialized for the child process } } ``` Fix. Use `@runInSeparateProcess` only when you need true global-state isolation (testing autoloader behavior, INI changes). For cross-process tests, build dependencies in the test method body, not in `setUp`, and avoid storing closures or database handles on `$this`. ``` /** @runInSeparateProcess */ class ConfigTest extends TestCase { public function testReadsConfig(): void { $factory = fn (string $k) => Config::get($k); $this->assertSame('value', $factory('key')); } } ``` With Mergify. Test Engine groups serialize-related failures distinctly from logic failures. The dashboard surfaces tests that fail with \`SerializationException\` so the @runInSeparateProcess scope decision is the obvious place to look. Pattern 8 ### PHPUnit retry attribute hiding real bugs Symptom. Your CI is green. A user reports a bug your tests should have caught, and the test report shows the failing test passed on attempt 3. Root cause. Plugins like [phpunit-retry](https://github.com/sebastianbergmann/phpunit/issues/3823) add a `@retryAttempts(3)` annotation that reruns failing tests up to N times. A real race that loses on attempt 1 and wins on attempt 2 gets reported as PASSED. The bug is still there. The pipeline has decided not to look at it. ``` use jamesheinrich\phpunit_retry\RetryTrait; class CheckoutTest extends TestCase { use RetryTrait; /** @retryAttempts 3 */ public function testChargesCard(): void { // intermittent timing bug; passes 2 of 3 times } } ``` Fix. Do not retry at the framework level. When a test is genuinely flaky, fix it. When the fix takes longer than a session, quarantine it instead. That keeps the signal visible without blocking the merge queue. With Mergify. Test Engine reruns at the CI level with attempt-level result tracking. You see that a test passed on attempt 2 of 3, which is exactly the information \`@retryAttempts\` throws away. Quarantine kicks in once the pattern is clear. Detection ## How to detect flaky PHPUnit tests in CI PHPUnit emits JUnit-compatible XML reports with the \`--log-junit\` flag. Point Mergify at the XML output of every CI run and Test Engine builds a confidence score for every test on your default branch. PR runs are compared against that baseline. Anything inconsistent gets flagged in a PR comment before the author merges. mergify ci ``` # 1. Emit JUnit XML on every CI run vendor/bin/phpunit --log-junit junit.xml # Or for paratest: vendor/bin/paratest --log-junit junit.xml # 2. Upload the result (once, in CI) curl -fsSL https://raw.githubusercontent.com/Mergifyio/mergify-cli/main/install.sh | sh mergify ci junit-process junit.xml ``` Prevention ## How to prevent flaky PHPUnit tests from merging On every PR, Mergify reruns the tests whose confidence is below threshold, without PHPUnit retry plugins or paratest's \`--rerun-failures\` touching your config. The PR gets a comment naming the unreliable tests, their confidence history, and whether the failure on this PR is new or historical noise. Authors fix the real bugs before merge instead of re-running CI until it passes. Quarantine ## How to quarantine flaky PHPUnit tests without skipping them Once a PHPUnit test is confirmed flaky, Test Engine quarantines it. The test still runs in the suite, no \`$this->markTestSkipped()\` rewrite required, but its result no longer blocks merges or marks the pipeline red. When the pass rate on main recovers, quarantine lifts automatically and the test goes back to being load-bearing. Your CI ## Run it on your CI The PHPUnit setup for Test Engine on each CI, how its retries compare with the CI's own reruns, and what makes PHPUnit flake on that CI's runners. - [PHPUnit on GitHub Actions](https://mergify.com/learn/flaky-tests/github-actions/phpunit) ## Want to see which PHPUnit tests in your repo are already flaky? Works with PHPUnit's built-in \`--log-junit\` output or paratest. No extra plugins required. Setup takes under five minutes. [Book a discovery call](https://mergify.com/discovery) ## Frequently asked questions Why are my PHPUnit tests flaky in CI but pass locally? Your laptop and your CI runner differ in CPU count, parallel paratest worker count, and whether the worker shares a database with siblings. Tests that mutate static properties, leave Mockery expectations unclosed, or freeze Carbon's clock without resetting surface those issues only under CI's tighter resource budget. Reproduce locally with \`vendor/bin/paratest -p 4 --recreate-databases\` to push the failure into the open before pushing. How do I detect flaky PHPUnit tests? PHPUnit alone cannot tell flaky from broken since each run gives one data point per test. You need to run the same commit multiple times and compare results. Mergify Test Engine does that on every PR and on the default branch, scores each test, and surfaces the tests whose pass rate drops below a confidence threshold. Does PHPUnit retry plugins or paratest's \`--rerun-failures\` fix flaky tests? No, it hides them. A test that fails on attempt 1 and passes on attempt 2 is still broken; you have only decided not to look at the failure. Use PHPUnit retry plugins or paratest's \`--rerun-failures\` as a temporary bandage for a test you are actively fixing, never as a permanent policy. For visibility without blocking the merge queue, quarantine instead of retry. What causes flaky tests in PHPUnit? Eight patterns cover most of what we see: static property leakage between tests, RefreshDatabase against parallel paratest workers, dataProvider returning shared mutable state, Mockery expectations leaking across tests, Carbon::now() without setTestNow, Guzzle MockHandler queues that outlive their test, processIsolation traps for global state, and PHPUnit retry attribute hiding real bugs. Each is covered above with a minimal reproducer. How do I quarantine a flaky PHPUnit test without deleting it? Mergify Test Engine quarantines the test automatically once its confidence score drops. The test still runs in the suite, but a failing result no longer blocks merges and its noise no longer drowns out real signal. When the test stabilizes on main, quarantine lifts automatically. No \`$this->markTestSkipped()\`, no commented-out tests, no orphaned files. How do I run PHPUnit + Laravel tests in parallel without database conflicts? Use \`paratest --recreate-databases\` so each worker gets its own database (or schema) and no two workers race on the same connection. Inside Laravel, \`RefreshDatabase\` works correctly per worker because each worker holds its own transaction. For SQLite, switch the test database to a named per-worker file via the \`DB\_DATABASE\_TEST\` placeholder; for Postgres/MySQL, the \`paratest\` runner appends the worker token to the database name. ## Ship your PHPUnit suite green. Mergify merges more than half a million pull requests a month without breaking main. [Get started](https://dashboard.mergify.com/)[Read the docs](https://docs.mergify.com/test-engine/) --- Source: https://mergify.com/learn/flaky-tests/playwright [Learn](https://mergify.com/learn)/[Flaky tests](https://mergify.com/learn/flaky-tests)/Playwright # Flaky tests in Playwright. Named, fixed, and quarantined. Flaky Playwright suites are not random. They follow patterns: auto-wait racing re-renders, route handlers too late, networkidle that lies, locator strict-mode surprises, headed-vs-headless drift. Name them, fix them, quarantine what is left. Your CI stays green. By [Rémy Duthu](https://www.linkedin.com/in/remy-d/), Software Engineer, CI Insights· Published April 2026 Example PR comment from the Mergify bot detecting a flaky Playwright test and quarantining it automatically. TL;DR The eight patterns behind nearly every flaky Playwright suite: auto-wait racing element re-renders, route handlers registered after goto, networkidle that never settles in SPAs, waitForResponse subscribed too late, locator strict-mode violations, storageState leakage, test.use() scope confusion, and headed vs headless drift. Each one has a minimal reproducer, a fix in stock Playwright, and a Mergify Test Engine detection strategy below. ## Why are Playwright tests flaky? Playwright's killer feature is [actionability auto-waiting](https://playwright.dev/docs/actionability): every interaction waits for the target element to be visible, stable, enabled, and receiving events before firing. That removes most of the explicit-sleep flakes that plagued Selenium and Cypress. It also creates a new flake surface, because "actionable" is computed against a real browser whose state is changing while the test runs. Add the rest of the browser-test stack and the surface grows. Service workers and websockets can keep [network idle](https://playwright.dev/docs/api/class-page#page-wait-for-load-state) from ever firing. Routes registered after `page.goto` miss the initial requests. `waitForResponse` subscribes too late if you call it after the action that triggered the request. Storage state shared across tests turns into ordering coupling. Tests that pass headed on your laptop fail headless in CI for reasons that come down to viewport size and font rendering. The patterns are finite. We've seen the same eight on Mergify Test Engine across hundreds of Playwright suites: auto-wait racing element re-renders, route handlers registered after page.goto, networkidle that never settles in SPAs, waitForResponse subscribed too late, locator strict-mode violations, storageState leakage across tests, test.use() scope confusion, and headed vs headless drift. Each has a clean fix once you can name it. ## The 8 patterns behind most flaky suites On this page 1. [01Auto-wait racing a re-render](#pattern-auto-wait-racing-a-re-render) 2. [02route() registered after goto()](#pattern-route-registered-after-goto) 3. [03networkidle that never settles](#pattern-networkidle-that-never-settles) 4. [04waitForResponse subscribed too late](#pattern-waitforresponse-subscribed-too-late) 5. [05Locator strict-mode violations](#pattern-locator-strict-mode-violations) 6. [06storageState leakage across tests](#pattern-storagestate-leakage-across-tests) 7. [07test.use() scope confusion](#pattern-test-use-scope-confusion) 8. [08Headed vs headless drift](#pattern-headed-vs-headless-drift) Pattern 1 ### Auto-wait racing a re-render Symptom. A click intermittently fails with \`Element is not attached to the DOM\` or \`Element is outside of the viewport\`, even though the locator matched a real element a moment ago. Root cause. Playwright's [actionability checks](https://playwright.dev/docs/actionability) wait for an element to be visible, stable, and receiving events. The check runs once and the action fires. If your component re-renders mid-flight (a state update lands, a CSS transition completes, a parent unmounts and remounts), the element your locator resolved is replaced before the click reaches it. ``` // Component swaps between