Flaky Test Tools Compared

Mergify Test Engine, Trunk Flaky Tests, Datadog Test Optimization and BuildPulse all detect flaky tests. What sets them apart is what happens next: whether the failure is handled inside the job, before the merge, and inside your merge queue.

Last updated: October 21, 2026

Start a free 14-day trial See Test Engine

The short answer

Mergify Test Engine detects and fixes flaky failures from inside your test framework, on any CI, and it ships with the merge queue.

Retried inside the job. A known flaky test is retried right where it failed, so the build goes green without anyone clicking rerun. Trunk can only quarantine the test after it has already failed your build.
Blocked on the pull request. Prevention reruns the tests a pull request adds or changes and stops the merge if one of them is flaky.
Quarantine that runs itself. A test goes into quarantine once the evidence holds and comes out when it is healthy again.
Built for the merge queue. Queue runs get the same retries and quarantine, so a flaky test never dequeues a pull request, and Fail First bisection finds the culprit fast when a batch really breaks.
One seat price. Merge Queue, CI Insights and Test Engine together, with no metering on test volume.

Feature comparison

From each vendor's own documentation and pricing page. Sources are listed at the bottom of the page.

Flaky test tools compared: Trunk Flaky Tests, Datadog Test Optimization, BuildPulse and Mergify Test Engine
Feature	Trunk Flaky Tests	Datadog Test Optimization	BuildPulse	Mergify Test Engine
What counts as flaky	Monitors you configure: pass-on-retry on the same commit, failure rate, failure count	Passed and failed on the same commit	Passed and failed on the same code, plus a failure-rate threshold (30% by default)	Passed and failed on the same commit. Every test is healthy, flaky or broken, so a real regression is never filed as noise
Retries a flaky test inside the job	No. Trunk works from the results you upload and acts through quarantine	Auto Test Retries reruns any failing test up to 5 times by default, real failures included	No	Yes. A known flaky test is retried right where it failed, inside your test framework, so the job goes green without waiting for quarantine
Catches a new flaky test on the pull request	No. Trunk reads the results you upload. Its new-test monitor only labels a test it has not seen before	Early Flake Detection retries new tests up to 10 times. A PR Gate can block the merge	Not documented	Prevention reruns the tests a pull request adds or changes, on the same commit, and blocks the merge if one is flaky
Quarantine	Automatic for tests already flagged flaky, or manual. The Trunk CLI rewrites the job's exit code when only quarantined tests failed	Manual, or by policy on time, branch or failure rate	Automatic or manual. Your pipeline reads the Quarantine API and skips those tests itself	Automatic once the evidence holds, so one bad infra run does not quarantine half your suite. Released automatically when the test is healthy again. Quarantined tests keep running on any branch
Merge queue	Trunk Merge Queue, included in the same plans	None. PR Gates cannot be required on GitHub repos that use a merge queue	None	Mergify Merge Queue, same plan. Queue runs get the same retries and quarantine, and Fail First bisection isolates a real failure fast
CI providers	14 named, including GitHub Actions, GitLab, Jenkins, CircleCI and Buildkite, plus others	Any CI provider	8 named, including GitHub Actions, CircleCI, GitLab CI and Jenkins (both beta), plus others	Any CI. Native GitHub Actions, Buildkite and Jenkins integrations, everything else through the Mergify CLI
How results get in	JUnit XML upload through the Trunk CLI, with guides for 28 frameworks	Datadog libraries for .NET, Java, JavaScript, Python, Ruby, Swift and Go, or JUnit XML upload	JUnit XML upload, with guides for 8 frameworks	Native in your test framework: pytest, Jest, RSpec, Vitest, Playwright, Go, JUnit and more (15 frameworks documented)
Ticketing	Jira, Jira Data Center, Linear and Asana, plus webhooks	Jira issues and Datadog work items	Jira, Linear, GitHub Projects (beta)	Jira and Linear coming. Slack notifications and an event log today
AI agents	Beta behind a waitlist. Trunk investigates, a Cursor Automation opens the PR	Bits Code generates a fix and can open a pull request	None. Owners get failure patterns, frequency data and linked commits	The Mergify CLI gives your coding agent each test's health and quarantine state, so it fixes the right tests
Pricing model	Per committer, metered on test spans. Free up to 5 committers and 1M test spans per committer. Team price not published	Per committer, from $20 a month billed yearly ($29 on demand)	By test volume, from $99 a month for 3M tests. Included with BuildPulse Runners	Per active seat, $21 a month, with Merge Queue and CI Insights in the same plan. No test-volume metering. Free up to 5 contributors

What counts as flaky

Trunk Flaky Tests: Monitors you configure: pass-on-retry on the same commit, failure rate, failure count
Datadog Test Optimization: Passed and failed on the same commit
BuildPulse: Passed and failed on the same code, plus a failure-rate threshold (30% by default)
Mergify Test Engine: Passed and failed on the same commit. Every test is healthy, flaky or broken, so a real regression is never filed as noise

Retries a flaky test inside the job

Trunk Flaky Tests: No. Trunk works from the results you upload and acts through quarantine
Datadog Test Optimization: Auto Test Retries reruns any failing test up to 5 times by default, real failures included
BuildPulse: No
Mergify Test Engine: Yes. A known flaky test is retried right where it failed, inside your test framework, so the job goes green without waiting for quarantine

Catches a new flaky test on the pull request

Trunk Flaky Tests: No. Trunk reads the results you upload. Its new-test monitor only labels a test it has not seen before
Datadog Test Optimization: Early Flake Detection retries new tests up to 10 times. A PR Gate can block the merge
BuildPulse: Not documented
Mergify Test Engine: Prevention reruns the tests a pull request adds or changes, on the same commit, and blocks the merge if one is flaky

Quarantine

Trunk Flaky Tests: Automatic for tests already flagged flaky, or manual. The Trunk CLI rewrites the job's exit code when only quarantined tests failed
Datadog Test Optimization: Manual, or by policy on time, branch or failure rate
BuildPulse: Automatic or manual. Your pipeline reads the Quarantine API and skips those tests itself
Mergify Test Engine: Automatic once the evidence holds, so one bad infra run does not quarantine half your suite. Released automatically when the test is healthy again. Quarantined tests keep running on any branch

Merge queue

Trunk Flaky Tests: Trunk Merge Queue, included in the same plans
Datadog Test Optimization: None. PR Gates cannot be required on GitHub repos that use a merge queue
BuildPulse: None
Mergify Test Engine: Mergify Merge Queue, same plan. Queue runs get the same retries and quarantine, and Fail First bisection isolates a real failure fast

CI providers

Trunk Flaky Tests: 14 named, including GitHub Actions, GitLab, Jenkins, CircleCI and Buildkite, plus others
Datadog Test Optimization: Any CI provider
BuildPulse: 8 named, including GitHub Actions, CircleCI, GitLab CI and Jenkins (both beta), plus others
Mergify Test Engine: Any CI. Native GitHub Actions, Buildkite and Jenkins integrations, everything else through the Mergify CLI

How results get in

Trunk Flaky Tests: JUnit XML upload through the Trunk CLI, with guides for 28 frameworks
Datadog Test Optimization: Datadog libraries for .NET, Java, JavaScript, Python, Ruby, Swift and Go, or JUnit XML upload
BuildPulse: JUnit XML upload, with guides for 8 frameworks
Mergify Test Engine: Native in your test framework: pytest, Jest, RSpec, Vitest, Playwright, Go, JUnit and more (15 frameworks documented)

Ticketing

Trunk Flaky Tests: Jira, Jira Data Center, Linear and Asana, plus webhooks
Datadog Test Optimization: Jira issues and Datadog work items
BuildPulse: Jira, Linear, GitHub Projects (beta)
Mergify Test Engine: Jira and Linear coming. Slack notifications and an event log today

AI agents

Trunk Flaky Tests: Beta behind a waitlist. Trunk investigates, a Cursor Automation opens the PR
Datadog Test Optimization: Bits Code generates a fix and can open a pull request
BuildPulse: None. Owners get failure patterns, frequency data and linked commits
Mergify Test Engine: The Mergify CLI gives your coding agent each test's health and quarantine state, so it fixes the right tests

Pricing model

Trunk Flaky Tests: Per committer, metered on test spans. Free up to 5 committers and 1M test spans per committer. Team price not published
Datadog Test Optimization: Per committer, from $20 a month billed yearly ($29 on demand)
BuildPulse: By test volume, from $99 a month for 3M tests. Included with BuildPulse Runners
Mergify Test Engine: Per active seat, $21 a month, with Merge Queue and CI Insights in the same plan. No test-volume metering. Free up to 5 contributors

Why teams pick Mergify

And what you give up with each of the others.

Choose Mergify Test Engine when

You want a flaky failure handled while the job is still running. Test Engine retries a known flaky test inside your test framework, so the build stays green without anyone clicking rerun.
You want new flaky tests stopped at the pull request. Prevention reruns the tests a pull request adds or changes and blocks the merge if one of them is flaky.
Flaky failures keep kicking pull requests out of your merge queue. Queue runs get the same retries and quarantine, and Mergify Merge Queue comes in the same plan.
You want quarantine to run itself: it waits for real evidence before quarantining a test and lets it out once it is healthy again.
You want one seat price for the merge queue, CI job health and test health, with no metering on test volume.

How Test Engine works

Datadog Test Optimization: what you give up

Auto Test Retries reruns any failing test up to 5 times by default. A real bug that passes on one of those retries goes green.
A PR Gate cannot be made a required check on a GitHub repository that uses a merge queue, and there is no merge queue of its own.
Billing is per committer, on top of the rest of your Datadog contract.

Mergify vs Datadog CI Visibility

Trunk Flaky Tests: what you give up

Trunk never retries a test itself. A flaky failure fails the job until the test is quarantined, so the first hits land on your developers and your merge queue.
It never reruns a new test either, so a new flaky test that passes once on its pull request merges unnoticed.
Quarantine works by having the Trunk CLI rewrite the exit code of every job that runs tests.
Usage is metered on test spans: the free plan stops at 5 committers and 1M spans per committer, and the paid price is not published.

Mergify vs Trunk

BuildPulse: what you give up

Quarantine is a list served by an API: skipping those tests is wired into your framework by you.
No free plan, only a 14-day trial, and pricing climbs with test volume. No merge queue.

Where Mergify is different

Test Engine runs inside your test framework, so it can act on a flaky failure while the job is still running. A test it already knows is flaky gets retried on the spot and the build stays green. Trunk and BuildPulse only see the results after the job has ended, so all they can do is quarantine the test once it has failed someone's build. Datadog retries inside the job too, but it retries every failing test, which also turns real bugs green when they pass on a second try.

New flaky tests never get that far. Prevention reruns the tests a pull request adds or changes on the same commit and blocks the merge if one of them is flaky, while the author still has the context to fix it.

For the rest, quarantine runs itself. Test Engine waits until a test has been unhealthy for a while, and ignores a run that turned many tests red at once, so an infrastructure hiccup does not quarantine half your suite. Once the test is healthy again it can leave quarantine on its own. All of it applies to merge queue runs, so with Mergify Merge Queue a flaky test never throws a pull request out of the queue.

And it is one plan. Merge Queue, CI Insights and Test Engine share a per-seat price with no metering on test volume. For the merge queue side of the market, see how Mergify compares.

Frequently asked questions

What is the best tool to detect flaky tests?

Mergify Test Engine, because detection and remediation both happen inside your test framework. A known flaky test is retried in the same job, so the build stays green. A new flaky test is caught on its pull request and blocked before it merges. Whatever is left is quarantined automatically once the evidence holds and released when it is healthy again. It works on any CI, and Merge Queue and CI Insights come in the same seat price. Trunk and BuildPulse only work from the results you upload, and Datadog retries every failing test, real failures included.

How do I quarantine flaky tests in CI?

All four tools quarantine, and the difference is how much work it leaves to you. Mergify quarantines inside your test framework on any branch, decides on its own once the evidence holds, and releases the test when it is healthy again. Trunk's CLI rewrites the job's exit code when only quarantined tests failed. Datadog's libraries keep the test running but stop its failure from affecting CI status. BuildPulse serves a quarantine list through an API and your pipeline has to skip those tests itself. See how to detect and quarantine flaky tests.

Which tools catch a flaky test before it merges?

Mergify and Datadog rerun new tests on purpose. Mergify's Prevention reruns the tests a pull request adds or changes, on the same commit, and blocks the merge if one is flaky, with or without a merge queue. Datadog's Early Flake Detection retries a test it has never seen up to 10 times, but its PR Gate cannot be required on a GitHub repository that uses a merge queue. Trunk and BuildPulse only read the results you upload, so they see flakiness when your CI happens to retry a test.

Do flaky test tools work with JUnit XML?

Yes. Mergify, Trunk and BuildPulse all take JUnit XML, so any framework that can write it is covered, and Datadog accepts it for test visibility.

Do I need a merge queue to use a flaky test tool?

No. All four work without one. Flaky tests just do the most damage in a queue, where one bad run can throw out a whole batch of pull requests. Mergify and Trunk both sell a merge queue next to their flaky test product. Datadog and BuildPulse do not. See how the merge queues compare.

What's a good tool to manage my tests?

If managing tests means knowing which ones you can trust, the four tools on this page are that category: they collect every test result from CI, separate flaky from broken, quarantine the noisy ones and track them until someone fixes them. Test case management tools for manual QA plans solve a different problem.

How much do flaky test tools cost?

Trunk bills per committer and meters test spans, free up to 5 committers and 1M spans per committer, with no published paid price. Datadog bills per committer, from $20 a month billed yearly. BuildPulse bills by test volume, from $99 a month for 3 million tests, and includes detection with its CI runners. Mergify bills $21 per active seat per month for Merge Queue, CI Insights and Test Engine together, with no metering on test volume, free for up to 5 contributors. See Mergify pricing.

Sources

Every claim above comes from the vendor's own documentation or pricing page.