---
title: "How to stop flaky tests from blocking your merge queue"
description: "Keep a flaky test from stalling your merge queue: anti-flake queue settings you can copy, automatic job retries, and quarantine for the tests that keep lying."
url: https://mergify.com/guides/stop-flaky-tests-blocking-merge-queue
---

Guide

# How to stop flaky tests from blocking your merge queue

Stop flaky tests from blocking your merge queue in three moves: make the queue tolerant of transient batch failures with `skip_intermediate_results`, retry jobs that failed for a known-flaky reason, and quarantine the tests that keep lying. This guide sets up all three.

By [Julien Danjou](https://mergify.com/about), Co-founder & CEO of MergifyUpdated July 29, 2026

## In one paragraph

Commit the `.mergify.yml` below so a failing batch that a larger passing batch contradicts stops dequeuing good pull requests. Add an Auto-Retry rule so jobs that fail on known flakiness re-run themselves. Then upload your test reports and quarantine your worst offenders, so a test that fails half the time on the same commit records the failure without blocking the merge.

## Why does one flaky test stall the whole queue?

Outside a queue, a flaky test costs you one rerun. Inside one, the cost compounds. The queue tests pull requests in batches, so when a batch goes red, Mergify assumes the batch is guilty: it cancels the batches behind it, puts those pull requests back in the queue, and splits the failed batch to find the culprit. That is exactly the right behavior when a real bug is hiding in there, and exactly the wrong tax to pay for a test that flipped a coin.

Meanwhile the author of the pull request at the front sees a failure on code that did nothing wrong and starts debugging it. That hour is the real cost, and it repeats every time the test misfires.

One setting will not cover it. Flakiness shows up at three altitudes (a batch, a job, a single test) and each one has its own answer. For what makes tests flaky in the first place, and framework-specific fixes, see the [flaky tests guide](https://mergify.com/learn/flaky-tests).

## Which layer handles which failure?

| Layer | What it handles | Where you set it | What happens |
| --- | --- | --- | --- |
| Queue anti-flake | A batch that fails once, then passes in a larger batch containing the same commits. | `.mergify.yml` | The pull request stays in the queue and merges. Nobody gets paged. |
| Auto-Retry | A whole job that failed on a test Mergify already flagged as flaky, or on a transient error you can match in the logs. | `Dashboard, CI Insights → Auto-Retry` | The workflow re-runs the failed jobs before a human ever looks at the red check. |
| Quarantine | One test that keeps producing different results on the same commit. | `Dashboard, or mergify tests quarantines add` | The test still runs, but its failure no longer fails the check or blocks the merge. |

Set them up in that order. The queue setting takes two minutes and pays off immediately, Auto-Retry needs a rule you tune once, and quarantine needs data before it can tell you what to quarantine.

## How do you stop a flaky batch failure from dequeuing good pull requests?

Turn on `skip_intermediate_results`. With it, Mergify stops requiring every intermediate batch to pass: if a later, larger batch containing the same changes goes green, the earlier red batch is treated as transient and the pull request still merges.

```
merge_queue:
  skip_intermediate_results: true
  max_parallel_checks: 3

queue_rules:
  - name: default
    merge_conditions:
      - check-success=ci
    batch_size: 5
    batch_max_failure_resolution_attempts: 3

pull_request_rules:
  - name: Queue approved pull requests
    conditions:
      - "#approved-reviews-by>=1"
      - check-success=ci
    actions:
      queue:
        name: default
```

Replace `check-success=ci` with the name of your own CI check before you commit this.

This is safe because a genuine bug also fails the larger batch that carries the same code, and that failure still ejects the pull request. Only the contradicted failures get skipped. Two caveats: it relies on the strict ordering of serial mode, so it is unavailable in [parallel mode](https://docs.mergify.com/merge-queue/queue-modes), and you may want it off while you are debugging CI and want to see every failure.

The other two keys shape how fast the queue recovers when a batch does fail. `max_parallel_checks` sets how many checks the queue runs at once, and with it how many pieces a failed batch gets split into, so a bisection that would take four sequential rounds finishes in one or two. `batch_max_failure_resolution_attempts` caps how deep that splitting goes before Mergify gives up and ejects the batch. If you are new to batches, the [batching explainer](https://mergify.com/learn/merge-queue/batching) covers the mechanics, and the [batch documentation](https://docs.mergify.com/merge-queue/batches) has the full split algorithm.

Do not have a queue yet? Start with [how to set up a merge queue on GitHub](https://mergify.com/guides/setup-github-merge-queue), then come back and add these keys.

## How do you retry a job that failed for a known-flaky reason?

Every CI has a rerun button. It works fine for one engineer hitting refresh and not at all for a queue processing hundreds of pull requests a day. [CI Insights](https://mergify.com/product/ci-insights) Auto-Retry does it on rules instead of on attention: go to CI Insights → Auto-Retry in the dashboard, pick your repositories, and add a rule.

A rule matches on the attributes of the failed job. The ones that matter here:

-   **Flaky** set to true, so you only retry jobs Mergify has already classified as flaky. This is the conservative rule to start with, and the one worth having in every repository.
-   **Log pattern**, a regular expression matched against job logs and check-run annotations. Something like `timeout|connection refused` catches the infrastructure failures that have nothing to do with your code.
-   **Conclusion**, to separate a job that failed from one that timed out, plus job name, pipeline, branch, and repository filters when you want to keep the rule narrow.

Then set the number of retries and the mode: `job` re-runs the failed jobs, `pipeline` re-runs the whole workflow. Because of a GitHub API limitation the retry is triggered at the workflow level either way, but jobs that already passed are preserved rather than re-executed. Full options are in the [Auto-Retry documentation](https://docs.mergify.com/ci-insights/auto-retry).

Keep the conditions tight. A blind retry-everything rule hides real bugs that only fail occasionally. A rule that only retries failures you already flagged as flaky keeps the opposite property, which is the one you actually want: a failure that survives the retry means something.

## How do you quarantine a test so its failure stops blocking merges?

Some tests are past retrying. A test that passes and fails on the same commit is not reporting on your code any more, and every merge it blocks is a merge blocked for no reason. [Test Engine](https://mergify.com/product/test-engine) quarantines it: the test keeps running and its results keep being collected, but its failure no longer fails the check.

First, send your test results to Mergify. Create an application key with the `ci` scope, store it as `MERGIFY_TOKEN`, and wire the upload into the job that already runs your tests. The two details that matter for quarantine are `continue-on-error` on the test step, so the upload step decides the conclusion once quarantined tests are discounted, and `test_step_outcome`, so a crashed runner that produced an empty report is not read as a pass.

```
- name: Run tests
  id: tests
  continue-on-error: true
  run: pytest --junit-xml=test-results.xml

- name: Upload test results
  if: success() || failure()
  uses: Mergifyio/gha-mergify-ci@v25
  with:
    token: ${{ secrets.MERGIFY_TOKEN }}
    report_path: test-results.xml
    test_step_outcome: ${{ steps.tests.outcome }}
```

Give it a couple of weeks of runs. The Detection page then lists your unhealthy tests, split into flaky (inconsistent on the same commit) and broken (failing consistently), each with a confidence level and an impact score counting the failed executions it caused. Sort by impact, take the handful at the top, and quarantine them from the dashboard or from the terminal:

```
mergify tests quarantines add -r owner/repo \
  --reason "flaky, tracked in MRGFY-1234" \
  "tests/auth/test_login.py::test_login_timeout"
```

The CLI form is handy in scripts, and it is the one to hand to an AI coding agent that just hit a flaky test and needs to get out of its own way. Once you trust the classification, turn on auto-quarantine per repository from the Mitigation page and Mergify quarantines newly detected flaky tests for you, with an extra option for known-broken ones. The [quarantine documentation](https://docs.mergify.com/test-engine/quarantine) covers the per-framework setup.

## How do you know it worked?

Watch the number of batch splits per day in the queue. That is the metric flakiness moves first, and the one that translates directly into CI minutes: fewer splits means the queue stopped chasing ghosts, and pull requests spend their time waiting on one CI run instead of four.

Then go back to the Mitigation page every couple of weeks and look at the quarantine list. Tests whose health improved can come out. Tests that have been quarantined for six months without anyone touching them are telling you something else, and Detection keeps them ranked by impact so you know which one to fix first. Quarantine buys time to fix the test properly. It is not where the test retires.

## FAQ

### Is quarantining a flaky test just ignoring the problem?

A flaky test is already lying to you: it says FAIL when the code is fine. Quarantine moves it from gating production to a to-fix list, and it keeps running so you still collect data. What makes it honest is that it is per test, never a global ignore-failures switch, and Detection keeps ranking it by how much CI time it wastes.

### Does skip\_intermediate\_results let a real bug through?

No. A real bug fails the larger batch that contains the same code, and that batch failure still removes the pull request from the queue. Only a failure that a passing superset batch contradicts is treated as transient. Turn it off if you need every intermediate state validated for compliance.

### Does it work with parallel queue mode?

skip\_intermediate\_results needs the strict cumulative ordering of serial mode, so it is not available in parallel mode. Auto-Retry and quarantine work in both, since they run at the CI level rather than in the queue.

### Do I have to change my test code?

No. You add an upload step to the workflow that already runs your tests, and Mergify reads the JUnit reports it produces. Test Engine supports pytest, Jest, Vitest, RSpec, Go, Rust, Playwright, Cypress, and a dozen more.

### How many tests should I quarantine to start?

Let detection collect a couple of weeks of data, sort by impact, and quarantine the handful at the top. Those few usually account for most of the queue disruption. Enable auto-quarantine once you trust the classification.

Last updated: July 29, 2026

## Keep the queue moving

Commit the config, upload your test results, and let Mergify tell the difference between a flaky test and a broken one. Free for up to 5 users.

[Get started](https://dashboard.mergify.com/)[See how Test Engine works](https://mergify.com/product/test-engine)